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ABSTRACT 


This project paper provides a systematic decomposition and industrial comparison 
of the U.S. defense decision-support process methodologies of Portfolio Analysis (PA). 
Included are current methods, tools, and models for ranking and evaluating strategic 
alternatives and options. The PA decision-support process analyzed in the study is used 
by the Department of Defense (DoD) to allocate resources to satisfy national strategic 
goals. Our working premise is that an effective decision-support process provides data 
driven knowledge directly to relevant decision makers to meet all U.S. defense and 
national strategic requirements, including proper balance of costs, risks, and capabilities 
in both routine operational and tactical battle space scenarios. 

To conduct this evaluation (4) Program Management (PM) and 2 PA tools were 
researched with a view to modeling identifiable strategic requirements juxtaposed with 
technical, financial, and implementation factors to meet today’s strategic capabilities 
requirements. What was learned is that together PA and PM tools used correctly provide 
useful tools for decision makers at all levels of government provided they are used 
correctly. While utilizing the modeling programs, experts in strategic and program 
analysis need to evaluate acquisition programs to rate the potential Estimated Military 
Value (EMV) in meeting the strategic needs of the United States. 
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EXECUTIVE SUMMARY 


This is a Portfolio Management Analysis assessment was conducted from August 
2008 through May 2009. The focus of this Joint Applied Project is identify and assess 
current Commercial Off-the-Shelf (COTS) Portfolio Analysis (PA) software products and 
solutions—with attention to market positioning, market share, product features, and other 
features. Two products (Risk Simulator and Palisades @Risk) were used to develop 
Portfolio Models. These models were populated with relevant data, and then run through 
an appropriate number simulation iterations to assess candidate projects with respect to 
risk and Expected Military Value (EMV). 

This document discusses Portfolio Management Analysis (PMA) during various 
stages of project management and system engineering. The goal for PMA is realized after 
the entire project design infrastructure is implemented from agency heads to managers 
and the end users instruments are provided for implementation. 

PMA requirements and scaling become secondary to the general strategic design 
and fit of each organization. Even where there is a lack of desired leadership, 
organizations should implement best practices and tools found in successful Portfolio 
Management Projects. This foundation enables organizational growth by providing 
design tools to achieve scalable requirements, operational readiness and the functions 
needed to provide sound balance business practices. However, broad PA implementation 
could more likely be slowed by policy, management and funding constraints, rather than 
technical considerations. 

We found that notwithstanding a need for some caution; PA tools are a way of 
enriching and equipping Department of Defense (DoD) communities with a means of 
professionally managing their resources. However, agencies and managers must balance 
new DoD Acquisition, Technology and Logistics Life Cycle Management Strategic 
Portfolio Development policies, as directed by Congress. Working PA tools at this level 
requires policy that ensures proper budget and program identification of existing or 
legacy, evolutionary or new development. The results of this analysis will be 
synthesized, documented and recommended to defense military organizations, and 



agency heads for consideration. The intent is to identify approaches and tools to 
incorporate PMA net-centric strategies to meet war fighter and business operations 
requirements, while continuing to maintain current levels of service, ensuring 
conservation of manpower and meeting infrastructure resource requirements. 

Key Recommendations 

1. Develop PA education programs. 

2. Develop a PA solutions and interoperability lab. 

3. Develop a PA Center of Excellence (CoE). The authors feel Naval Postgraduate 
School (NPS) is ideally suited to develop the curriculum, lab, CoE, and training 
programs. In short, NPS is well suited to provide a “PA Solution Center” within 
DoD. 
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I. U.S. DEFENSE PORTFOLIO ANALYSIS 

A. PORTFOLIO ANALYSIS 

This chapter defines and describes PA; where it started, what it measures, how 
other industries are utilizing it, why it is important today, and what the DoD is doing and 
planning for the future. 

Modem Portfolio Theory was introduced by Harry Markowitz with his paper 
“Portfolio Selection,” which appeared in the Journal of Finance (1952). He 
demonstrated that a portfolio of individual securities composed of consistently good risk- 
reward characteristics (e.g., stocks of all rail companies), could well be foolish. He 
detailed the mathematics of diversification, which focused on selecting portfolios based 
on their overall risk-reward characteristics. He felt that investors should create portfolios 
of dissimilar securities rather than purchase and hold only individual securities (e.g., only 
shares of IBM). Portfolio theory provides a broad context for understanding the 
interactions of systematic and non-systematic risk and reward. 

Non-systematic risk (diversifiable) is specific to a particular security or sector so 
that the impact on a diversified portfolio is limited. This indicates that DoD strategic 
decision makers must not place all their assets (securities) in a single asset type. The risk 
is that the DoD might, for example, plan an air war with hundreds of planes but fail to 
provide sufficient ground forces to follow up and finish the job. 

Systematic risk (non-diversifiable) cannot be eliminated by having a portfolio 
with many asset types. This risk affects the entire economy (or a sector of it), and is also 
known as market risk. This is important in evaluating economic risk when deciding 
which assets to include in the defense strategy. Risk has profoundly shaped how 
institutional portfolios are managed and has motivated the use of passive investment 
management techniques. 
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The mathematics of portfolio theory is used extensively in Financial Risk 
Management (Financial Risk Management, 2008) and was a theoretical precursor for 
today’s Value-at-Risk (VaR) (2009) measures (Portfolio Theory). 

PA is the art and science of allocating scarce resources to satisfy strategic 
objectives, or determining how to best spend limited dollars (Flynn & Field, 2006). PA 
also provides tools for organizing and managing a set of projects in a portfolio of projects 
to meet its goal (COCP, 2004). Portfolio Management begins with an enterprise-level 
identification and definition of market opportunities and then prioritization of those 
opportunities within resource constraints (GAO, 2007, p. 9). A set of projects tracked 
across the entire portfolio in a timely and effective manner helps senior leadership make 
sound decisions, data-based decisions supported by analysis of cost, schedule and 
performance risks. These future projects will have a National strategic impact as 
situations and partners change. The ability of senior leadership to adjust portfolios to 
meet defense needs now and in the future is critical. 

PA is used by businesses to measure everything from money to performance. In 
the finance industry, it is used to measure the strength of a group of investments to make 
appropriate tradeoffs of expected return on investment and risk. Using the Markowitz 
Efficient Frontier (MVO, 2009), a ratio of the expected return for each asset, the standard 
deviation of each asset’s logarithmic relative returns (measure of risk), and the 
correlation matrix between these assets, sets of portfolios with expected returns greater 
than any other with the same or lesser risk, and lesser risk than any other with the same or 
greater return could be identified (MVO, 2009). In the Information Technology (IT) 
sector, PA is used to manage priorities for resource allocation. Based on limited resources 
(budget), which projects should we keep while increasing profits and which are failing to 
perform and losing money? Whatever is being measured during the analysis, it is a key 
factor in the success or failure of the business. Companies commonly use Net Present 
Value (NPV) analysis, which can show, in today’s dollars, the relative cash flow of 
various alternatives over a long period of time (GAO, 2007, p. 15). 

In general, successful companies take a disciplined approach to prioritizing needs 
and initiating a balanced mix of executable development programs (GAO, 2007, p. 7). 
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They begin with an enterprise level approach to identifying market opportunities and then 
prioritize them based on strategic goals, resources available, and risk. The market 
opportunities with the greatest potential to succeed are included in the portfolio. 

IBM focuses on what it calls “industry integrated solutions”, which are portfolios 
structured around customers’ and buyers’ behaviors and needs and not on specific 
product offerings. IBM looks at the customers’ needs and the market segments that 
might provide for new opportunities. The “Strike Zone” is the core business for IBM. 
They then look at the “traditional” opportunities where current customers might be 
attracted to an existing business which might be enhanced. Then they consider “pushing 
the envelope” opportunities where they might attract new customers or create a new 
business. On the very outside would be the “white space opportunity” or the most risky 
business. This is where IBM might develop something new to the industry—based on 
new customers’ wants or needs. Based on available resources and potential profitability, 
a balanced portfolio across all segments is obtained (GAO, 2007, p. 10). 

Motorola targets market segments at the enterprise level to ensure a balanced mix 
of projects and resources is maintained. Their Government and Enterprise Mobility 
Solutions business unit uses a 70-20-10 formula where 70% of the projects and resources 
are for the core business, 20% in new markets with existing products, and 10% for 
discovering new markets and products (GAO, 2007, p. 11). 

So why is PA important today? The Clinger-Cohen Act of 1996 mandates its use 
for all federal agencies. The GAOs “Assessing Risk and Returns: A Guide for 
Evaluating Federal Agencies’ IT Investment Decision-Making,” Version 1, (Government 
Accountability Office, February 1997, GAO/AIMD-10.1.13) requires that IT investments 
apply Return on Investment (ROI) measures. DoD Directive 8115.01 (Department of 
Defense Directive, 2005, DODD 8115.01), issued October 2005, mandates the use of 
performance metrics based on outputs, with ROI analysis required for all current and 
planned IT investments. DoD Directive 8115.02 (Department of Defense Instruction, 
2006, DODI 8115.02) implements policy and assigns responsibilities for the management 
of DoD IT investments as portfolios within the DoD Enterprise, where they defined a 
portfolio to include outcome performance measures and an expected ROI. The DoD Risk 
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Management Guidance Acquisition guide book requires that alternatives to the traditional 
cost estimation be considered because legacy cost models tend not to adequately address 
costs associated with information systems or the risks associated with them (Mun & 
Housel, 2006, p. 1). The CJCSI 8410.01, (Chairman of the Joint Chiefs of Staff 
Instruction, 2007, CJSC 8410.01), establishes policies and procedures for the War 
fighting Mission Area Information Technology Portfolio Management and net-centric 
data sharing processes. 

Over the next several years, DoD plans to invest $1.4 trillion in major weapons 
system programs. Continued failure to deliver weapons systems on time and within 
budget not only delays providing critical capabilities to the war fighter, but results in less 
funding for other DoD and federal needs (GAO, 2007, p. 1). With this level of spending 
and an upcoming reduction in DoD obligation, it is important for the DoD to spend its 
money as efficiently as possible. This can only be accomplished by better evaluating the 
programs/systems for risk before they start being funded to truly ascertain their overall 
value toward meeting the strategic goals of the U.S. These programs contain 
considerable risks in the form of cost overruns, schedule delays and performance failures. 

So, what is the DoD currently doing? The DoD is using individual program 
managers to manage specific programs/systems, without regard to the overall strategic 
goal of the U.S. Each program is its own entity, with little or no interaction with other 
programs and program managers are not held responsible for minimizing the risks 
associated with their particular programs. The DoD’s service-centric structure and 
fragmented decision-making processes are at odds with the integrated, portfolio 
management approach used by successful commercial companies to make enterprise- 
level investment decisions (GAO, 2007, p. 18). 

In 2004, the Defense Finance and Accounting Service (DFAS) implemented 
portfolio management in an effort to help prioritize initiatives and more closely link 
budget to agency strategy, while answering a presidential call for improving financial 
management. In doing this, they developed an approach which not only governs 
technology investments but includes all high-value initiatives ($250,000 or more). As a 
decision-making tool, Portfolio Management requires essential data about all initiatives 
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to be entered into a central database and requires those initiatives to be scored against 
basic criteria and risk (decision analysis). Portfolio Management treats existing and new 
initiatives as assets to be managed instead of costs. The process is dynamic and iterative 
so that the Portfolio reflects changing agency goals and priorities. The key to assessing 
portfolio effectiveness is measuring the right things. Because of the importance of 
performance measures in completing the portfolio requirements, it is crucial for DFAS to 
agree on the appropriate measures early in the Portfolio Management process. Each 
initiative receives a weighted score on three dimensions: Mission, Financial and Benefits, 
and Risk (Hubbard, 2004). 

Using the Joint Capabilities Integration Development System (JCIDS), needs are 
identified, technologies are evaluated, milestone decisions are planned out and then 
program execution begins. The JCIDS process emphasizes early attention to the fiscal 
implications of newly identified needs, including the capability to divest the Department 
of lower priority or redundant capabilities. Despite these provisions, assessments of war 
fighting needs continue to be driven by the services and to be based on investment 
decision-making processes that do not function together to ensure that DoD pursues 
needs that its resources can support (GAO, 2007, p. 19). Another issue is that, unlike 
commercial enterprises which can make program decisions, the services and program 
managers can not. The DoD has functional capability boards that oversee each of eight 
functional areas. The boards lack the authority to allocate resources and to make or 
enforce decisions to divest their capability area of existing programs to pay for new ones 
(GAO, 2007, p. 21). 

B. PROJECT BACKGROUND 

According to former Secretary of Defense (SECDEF), Donald Rumsfeld, “What 
you measure, improves.” In this regard, the DoD is adept at measuring the cost and the 
value of a specific program to fulfill a specified mission. Trade-off studies are conducted 
and analyses of alternatives are performed. Sometimes, gap analyses are performed. But 
are such comparisons made program to program? Are funding decisions made with an 
emphasis on leadership strategy and national objectives? Portfolio analysis is a 
promising method to improve DoD business practices by analyzing a portfolio of systems 


5 



as a whole, rather than analyzing individual acquisition programs or cost. While the 
above statement is in fact true, PA can be utilized as low as the command level to 
evaluate programs and projects using real options, whether current or future, used or not, 
at no current cost to the command. These options can then be sent to the senior 
commands for further evaluation and placement into higher level options, all the way to 
the department/service level. Once at the department/service level, the decision makers 
have a better tool to evaluate these options against the national strategic plans in order to 
make a better-informed decision. 

In the U.S. Military context, risk analysis, real options analysis, and portfolio 
optimization techniques are enablers of a new way of approaching the problems of 
estimating ROI and the risk-value of various strategic real options (Mun & Housel, 2006, 
p. 1). This analysis can be performed by running various risk modeling programs 
including, Monte Carlo Simulations, Stochastic Forecasting, Real Options Analysis 
(ROA), and Portfolio Optimization using Knowledge Value Added (KVA). These 
methodologies help in making the best possible decisions, allocating budgets, predicting 
outcomes, creating portfolios with the highest strategic value and ROI, and so forth, 
where the conditions surrounding these decisions are risky or uncertain (Mun & Housel 
2006, p. 2; Davis, 2003). 

What are these modeling programs? Monte Carlo methods are a class of 
computational algorithms that rely on repeated random sampling to compute their results. 
Monte Carlo methods are often used when simulating physical and mathematical 
systems. Because of their reliance on repeated computation and random or pseudo¬ 
random numbers, Monte Carlo methods are most suited to calculation by a computer. 
Monte Carlo methods are useful for modeling phenomena with significant uncertainty in 
inputs, such as the calculation of risk in business. In Stochastic Forecasting, the objective 
is to minimize a given cost function that depends on a large number of discrete or 
continuous variables. ROA applies financial options valuation techniques to real physical 
assets and capital budgeting decisions (Campbell & Harvey, 2002). ROA itself is the 
right, but not the obligation, to undertake some business decision; typically the option to 
make, or abandon, a capital investment (Fintor, 2009). PA provides decision makers with 
an efficient set of portfolios, based on minimizing risk subject to a particular return 
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(Walls, 2004). Risk modeling refers to the use of formal econometric techniques to 
determine the aggregate risk in a financial portfolio. Risk modeling is one of many 
subtasks within the broader area of financial modeling. Risk modeling uses a variety of 
techniques including market risk, VaR, Historical Simulation, or Extreme Value Theory 
in order to analyze a portfolio and make forecasts of the likely losses that would be 
incurred for a variety of risks. Such risks are typically grouped into credit risk, liquidity 
risk, interest rate risk, and operational risk categories (Fintor, 2009). 

As part of the project background, we must include a short discussion about risk. 
So what is risk? Risk is any uncertainty that affects a system in an unknown fashion and 
its ramifications are unknown, but it brings great fluctuation in the value and outcome. 
Risk has a time horizon, meaning that uncertainty evolves over time, which affects 
measurable future outcomes and scenarios with respect to a benchmark (Mun & Housel, 
2006, p. 33). The DoD does not just budget for one year at a time. There are annual fully 
funded programs and multiyear funded programs. There are programs that have annual 
appropriations, bi-annual appropriations, and five-year appropriations. The question then 
is, how does the DoD plan for these out year budgets while accounting for the risk of 
program failure, increased cost, delays in scheduled deliveries, changes in strategic 
options and goals, or even the inadequacy of technological maturity? Every four years 
we elect a new administration, which creates its own foreign policy and strategic 
objectives. As these objectives change, so must the analysis of DoD’s portfolios and 
their ability to satisfy our national strategic objectives. One would hope that the 
budgeting process was one in which the DoD entered with its eyes wide open and with a 
full understanding of the risks involved. So what could the DoD be doing to minimize 
risk? First, instead of looking at the short term and making decisions based on current 
ROI or cost, they should look toward the long term investment risk-return ratio, total 
strategic value, to include returns, cost, strategic options, (options available now or later, 
whether taken or not), as well as risk (Mun & Housel, 2006) and reduction of program 
duplication across services. 

The Pentagon is in the midst of making investment decisions using so-called 
“capability groups” across the entire military, as opposed to managing them on a service- 
specific, program-specific manner. Enhancing the “portfolio management” initiative is 
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one of the items on SECDEF England’s list of goals he would like to accomplish by the 
end of this year. England launched the initiative in 2006 to group closely related military 
systems from all Services and DoD Agencies and manage them in groups. A similar 
mechanism has been established for Command and Control (C2) systems. The goal is to 
keep the military services from pursuing redundant and pricey platforms. In his February 
7th memo, England directed that four capability groups, tested during the construction of 
the Pentagon’s 2009 budget blueprint, be made permanent. Under those now-official 
groups, military programs from across the DoD will be grouped into these four categories 
to help with investment decisions: C2; Battle Space Awareness; Network-Centric 
Warfare (NCW); and Logistics (Bennett, 2008). 

Hopefully, the DoD master national strategy will help to guide investment 
decisions based on the entire national strategy and across all services rather than on the 
needs or wants of a single service, using capability requirements. This has been 
addressed as recently as September, 2008 with the release of DoD Directive (DODD 
7045.20, 2008), Capability Portfolio Management. The nine capability areas outlined in 
the DODD are: C2, Battle Space Awareness, NCW, Logistics, Building Partnerships, 
Protection, Force Support, Force Application, and Corporate Management and Support. 
The latter five are in the interim phase through Fiscal Year (FY) 2010 Program Objective 
Memorandum. The main policy point is stated as such: The DoD shall use capability 
portfolio management to advise the Deputy Secretary of Defense and the Heads of the 
DoD Components on how to optimize capability investments across the defense 
enterprise (both materiel and non-materiel) and minimize risk in meeting the 
Department’s capability needs in support of strategy (DODD 7045.20, 2008). 

We have talked about both PA and portfolio management a great deal up to this 
point. To clarify, this paper focuses on PA, but in doing so, we must also discuss 
portfolio management. There are similarities, such that in both portfolio (decision) 
analysis and portfolio management one must analyze and practice risk management. PA, 
using the Markowitz Efficient Frontier (MVO, 2009) optimization approach, provides 
decision makers with an efficient set of portfolios based on minimizing risk subject to a 
particular return. 


8 



Portfolio Management, on the other hand, provides guidance as to what level of 
risk-taking is appropriate. PA alone does not provide managerial guidance about which 
efficient portfolio is best for the organization. Combining PA and portfolio management 
can improve the overall decision process, and could ultimately improve organizational 
performance (Walls, 2004). This paper focuses on the first phase, Analysis, as depicted in 
Figure 1 below. 
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fighter 


Figure 1. Analysis Phase of Portfolio Management Process 


When the Honorable Richard Greco, Jr., Assistant Secretary of the Navy 
(Financial Management and Comptroller), entered his position in late 2004, he was asked 
by the Vice Chief of Naval Operations to consider devising methods that could be used to 
analyze programs and better inform resource decision making early in the Planning, 
Programming, Budgeting and Execution process. This request was the beginning of an 
effort to analyze common and best practices in government and in the private sector and 
then to use and augment those techniques in a Department of the Navy (DoN) construct. 
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A Mine Countermeasures (MCM) PA pilot was established and fifteen Subject 
Matter Experts (SMEs) were allowed to score forty different systems and six platforms 
using a model similar to that used at U.S. Special Operations Command, the Strategy-to- 
Task Model, which is an exemplary DoD model. In the MCM pilot, common criteria 
were established based on the strategic goals of the organization. In scoring capabilities 
and risks of MCM systems, a logical, rigorous, strategy-to-tasks approach is used, 
designed to link individual assets such as ship’s sonar, and influence-sweep sleds to 
broad-based, macro national security objectives. By employing this approach, current and 
proposed systems are evaluated in terms of their contribution to the goals and priorities 
set. 


C. PROJECT METHODOLOGY 

1. Problem Identification 

Dr. Flynn has already identified common problems with regard to existing DoD 
models and portfolio management techniques. Most existing DoD models fall short. For 
example, Analysis of Alternatives and campaign analyses are not PA. Other models and 
processes consider only a subset of tasks, costs, and capabilities and do not take a holistic 
view of the portfolio. Other common problems include (Flynn & Field, 2006): 

a. Portfolios do not reflect national strategy, 

b. Poor overall quality with regard to project selection, 

c. Lack of focus on the right projects; resources are wasted on the wrong 

projects, and 

d. Multi-year funding designed to save money and improve the defense 

industrial base not being appropriately utilized. 

Multi-year contracts are expected to achieve lower unit costs compared to annual 
contracts, through one or more of the following sources: 

a. Purchase of parts and materials in Economic Order Quantities, 

b. Improved production processes and efficiencies, 

c. Better utilized industrial facilities, 
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d. Limited engineering changes due to design stability during the multiyear 

period, and 

e. Cost avoidance by reducing the burden of placing and administering annual 

contracts. 

The use of multi-year contracts does not come without risks; such as increased 
costs to the government, should the multi-year contract be changed or canceled, and 
decreased annual budget flexibility for the program and across the DoD’s portfolio of 
weapon systems (GAO, 2008, p. 5). 

Part of our methodology will be to use the integrated risk analysis framework as 
outlined by Dr. Mun. The framework consists of eight distinct phases of a successful and 
comprehensive risk analysis implementation. The eight phases (Mun & Housel, 2006, 
pp. 22, 23) are: 

1. Qualitative Management Screening 

2. Time-series and Regression Forecasting 

3. Base Cost NPV Analysis 

4. Monte Carlo Simulation 

5. Real Options Problem Framing 

6. Real Options Modeling and Analysis 

7. Portfolio and Resource Optimization 

8. Reporting and Updating Analysis 

Using this framework, we will develop a sample scenario which will approximate 
a basic DoD program, with options over a set time period. Using this scenario we will 
test several COTS programs to evaluate which product, in fact, provides the best analysis 
for use in the DoD based on the eight phases of the risk analysis framework. 

2. Systems Engineering 

Systems Engineering is responsible for overseeing the COTS/Govemment Off- 
the- Shelf (GOTS) analysis and selection process and developing appropriate criteria for 
assessing and evaluating the applicability and fit of existing components pertaining to the 
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current development effort. Gathering and assessment of COTS/GOTS (C/G) product 
data may be performed by specific domain engineers or SMEs. There are new 
methodologies, which are helping to make the best possible decisions, allocate budgets, 
predict outcomes, create portfolios with the highest strategic value and ROI, and so forth, 
where the conditions surrounding these decisions are risky or uncertain (Mun & Housel, 
2006, p. 2). For this research paper, a Systems Engineering effort has been conducted 
using a COTS life cycle model. The COTS life cycle model produces a system primarily 
out of one or more available COTS products. This means that the system functionality 
being developed, the system performance and the operational characteristics are primarily 
driven by what is commercially available. Requirements and system objectives are 
developed to understand the functionality and performance characteristics, and are then 
used as requirements to select a set of COTS products that provide most of the system 
functionality and desired performance. Instead of relying purely on immediate ROI or 
cost, a project strategy, process innovation, or new technology, should be evaluated based 
on its total strategic value, including returns, cost, and strategic options, as well as risks 
(Mun & Housel, 2006, p. 2). 

3. COTS Integration Life Cycle 

The goal of the COTS Integration model is to build an end product that is 
primarily an integration of available COTS products. The outcome of this model is a 
product solution that will meet customer/user requirements. COTS products are identified 
and analyzed for use in the proposed solution. A main advantage of using the COTS 
Integration model is the speed at which a product can typically be rolled out to the 
customer/user. Also, by incorporating COTS products for certain functions, there is 
usually a high level of confidence that the end product will work according to 
expectations of the COTS products. However, not all requirements may be satisfied by 
the solution and it is important that the customer/user accepts the limitations that the 
COTS product(s) may impose. The customer may need to trade specific functionality or 
at least be able to be flexible with regard to the solution. For example, a COTS product 
may dictate a change to a customer process flow, or may require the customer to provide 
needed functionality from another source. The COTS Integration life cycle promotes a set 


12 



of activities that supports a customer’s need for implementing an 80/20 rule-type project 
(i.e., to produce a system with 80% of the desired functionality with 20% of the cost 
and/or schedule). 

To begin the analysis, capability requirements are developed in the form of 
product requirements. Next, available COTS products are identified and analyzed for use 
in the proposed end product. The capability requirements are then mapped to product 
features. While the process is simple, the spectrum of COTS PA products is broad, and 
the requirements are specific to DoD. The analysis needs to encompass the multi¬ 
dimensional aspects of product feature comparison and total-cost-of ownership. The 
work that Dr. Flynn has done in identifying attributes of PA has been used as 
requirements for PA product research and comparative analysis. Product literature, 
demonstrations, downloadable evaluations, vendor interviews, and objective product 
reviews will be used to plot each PA solution’s capabilities against the requirements 
provided by Dr. Flynn (Flynn & Field, 2006). 

4. COTS Analysis and Selection 

The C/G analysis and selection System Engineering and Integration process 
component considers a wide range of existing and available COTS and GOTS system 
products (ranging from individual hardware components or software algorithms or 
modules to large system components) for use in current system development. This 
process is based on the technical assumption that certain parts of large systems can be 
effectively and efficiently assembled through use of existing components and this use can 
potentially reduce development costs, enable rapid assembly and integration of systems, 
and reduce the maintenance costs associated with the support and future upgrade of the 
overall system. There are typically two major activities for this process: 

a. Analysis of current operational and support requirements, and 

b. Availability of C/G elements to meet requirements and selection of the 
appropriate C/G products to incorporate into the current system 
development effort 
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5. Appropriate Data 


According to Dr. Flynn, the government needs to implement PA to quantify and 
justify programming and budget decisions during the Planning, Programming, Budget, 
and Execution process (Flynn & Field, 2006). He asserts that, when armed with this 
analysis, senior leadership can alter projects, drop projects, or increase projects based on 
current threats and proposed future needs. Establishing a portfolio of projects that can be 
altered, based on the current world situation, best value for the government dollar and 
new technology, is a best business practice that other industries are already utilizing. Dr. 
Fly nn further asserts that the private sector uses a common metric (i.e., dollars) to 
determine value. 

There is no commonly defined metric for value across DoD programs. As a 
result, military value is extremely difficult to determine and must be subjective. Dr. 
Fly nn has provided information on the criteria and data that is required to conduct PA for 
the purpose of evaluating programs comparatively. He proposes replacing the Expected 
Commercial Value model (used in commercial contexts) with an EMV. The National 
Security Strategy of March 2005 identifies four strategic objectives and eight required 
operational capabilities of U.S. military forces. The EMV, in one of many constructs, is a 
function of the strategic importance of a project, the degree to which the capability is 
desired, as well as probabilities of technical and operational success. It is important to 
note that the variable for Strategic Importance is influenced directly by strategic 
objectives; similarly, the degree of capability desired is influenced by key operational 
capabilities. 

6. Life Cycle Activities 

Engineering activities in the COTS Integration life cycle begin primarily with an 
iterative interaction between the Requirements and Concept Development, Analysis and 
Initial Design, Architecture Definition and C/G Analysis and Selection components. 
Within Requirements and Concept Development a set of initial system objectives are 
elicited and compiled into a product vision or feature list. This information is then used 
to identify and collect information about existing products that could satisfy customer 
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needs. Trade Studies represent a key technical management component that supports the 
life cycle activities. Among the greatest risks in a C/G-type project are the actual 
performance of the selected products and the performance of C/G products that may be 
integrated and interfaced in ways that were previously not considered. Trade Studies, 
conducted within the Decision Analysis and Resolution component, using a set of 
potential C/G products executing in realistic scenarios, can significantly reduce this risk 
to the project. 

7. Design, Integration and Evaluation Activities 

Once the prospective COTS products have been identified, Detailed Design and 
Verification activities can commence. Detailed Design is focused on the system 
capabilities that are needed to enable the selected COTS products to interact with each 
other and/or the user (e.g., glue components). Therefore, it should represent a very 

minimal activity. It is conceivable that Detailed Design may not even be needed (e.g., a 

single COTS system is selected to provide the entire functionality). Early Verification 
activities focus on demonstrating that the chosen C/G products actually perform as 
anticipated on the project and support the required interfaces. Verification of the 
components can be thought of as “unit tests” for the components that are going to be 
integrated into the delivered system. 

8. System Engineering Activity Descriptions 

a. Perform Initial Analysis of C/G Elements 

Our research team will perform initial analysis of the C/G elements, 
including identifying the screening criteria for evaluation of GOTS/COTS candidates and 
evaluating the properties of candidate C/G components. Such properties include 
component functionality (what services are provided) and other aspects of a component’s 
interfaces (e.g., standards implemented, operational environment). These properties also 
include quality and supportability aspects of the components that may be more difficult to 
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isolate and quantify than component functional and performance aspects. Based on the 
screening, a short list of candidate C/G components will be identified for use in the next 
activity. 


b. Perform Literature Review 

Our research team will collect information on PA solutions to obtain case 
studies, and industry best practices. The following sources will be used: 

• DoD components and defense firms 

• Journal articles 

• Academic and Wall Street SMEs 

• Periodicals and magazines which conduct product comparisons 

• Other industries utilizing PA 

c. Select Appropriate C/G Elements for Inclusion in Current 
System Development 

Our research team will determine the “fitness for use” of existing C/G 
components that are candidates for application in the new system development effort. 
We will then determine which available C/G components represent a “best fit” for the 
current development effort. It is also reasonable to discover “non-technical” component 
properties, such as the vendor’s market share, past business performance, and process 
maturity of the developing organization. The selection process may also extend to 
include qualification of the development process used to create and maintain candidate 
C/G products. The determination on use of existing C/G products may be dictated by the 
acquirer or another appropriate stakeholder. Thus, this process may actually be executed 
in advance of completion of the Requirements and Concept Development process since 
the outcome of the C/G analysis and selection may impact the development and 
definition of system requirements. 
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II. SYSTEM ENGINEERING (SE) 


A. REQUIREMENTS 

1. Objectives 

SE begins with the capture of requirements. For this research project, several 
sources were used to establish requirements. In understanding the requirements provided 
from different organizations, it is necessary to understand how each organization defines 
PA. The Project Management Institute defines: 

Portfolio as “A collection of projects and programs and other work that are 
grouped together to facilitate effective management of that work to meet strategic 
business objectives. The projects or programs of the portfolio may not necessarily be 
interdependent or directly related.” 

Portfolio Management (PfM) as “The centralized management of one or more 
portfolios, which includes identifying, prioritizing, authorizing, managing, and 
controlling projects, programs and other related work to achieve specific strategic 
business objectives.” 

PfM, in the context of this research paper, views each program investment from 
an enterprise level as contributing to the collective whole, rather than an independent and 
unrelated program investment. With this enterprise perspective, Portfolio Managers can 
effectively: 

a. Identify and prioritize opportunities and 

b. Apply available resources to potential products to select the best mix of 
products to exploit the highest-priority and/or most promising— 
opportunities (GAO, 2007) 

This type of approach depends on strong enterprise governance with committed 
leadership; it also depends on information tools which provide the ability to visualize 
complex data relationships in a comprehensible manner. 
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As depicted in Figure 2, a PfM approach begins with an enterprise-level 
framework and definition of market opportunities and then the prioritization of those 
opportunities within resource constraints. At each review gate, programs are assessed 
against available resources, established criteria, competing programs, and the goals and 
objectives of the DoD as a whole. As alternatives pass through each review gate, the 
number is expected to decrease, until only those alternatives with the greatest potential to 
succeed make it into the product portfolio (GAO, 2007). 



Figure 2. PfM Approach to Product Investments (From GAO, 2007) 


In support of the framework identified as a best practice by the GAO (GAO, 
2007), a fundamental premise of this paper is the need to identify COTS systems that can 
provide the capabilities identified in the Requirements section. 

This project analyzes applicable PA tools, systems and underlying methodologies 
in terms of relevancy to identification of viable requirements and technical alternatives. 
As depicted in Figure 3, the PA activities include: 

• Identifying and Maintaining the Baseline 
o Collecting data 

o Grouping Investments into Capability Areas 


18 
































Identifying Gaps 
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Figure 3. Portfolio Analysis (PA) Activities (From Mean-Variance Optimization, 2009) 


2. Scope 

This research project encompasses PA pertaining to the DoD, as identified in the 
requirements provided, both explicit and derived. Some interpretation of requirements is 
needed as requirements are decomposed into functional capabilities. These 
interpretations, in terms of this paper, have not been presented to, nor vetted with the 
authors and agencies that provided the top-level requirements. The research scope fully 
explores the applicability of PA systems, vis-a-vis the requirements, using a System 
Engineering approach. 
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This report focuses on identifying current COTS PA systems, operational 
constraints of current systems, and general recommendations for system 
implementations. The technology evaluated includes PA software, computers and related 
technology. The approach includes gathering the information needed (requirements, best 
practices) to conduct COTS product evaluations. 

3. Identified Needs and Best Practices 

This section begins to address the research questions in terms of performance, 
functional requirements, and system requirements. As outlined in the Project 
Methodology, a Literature Review was conducted to examine the current PA landscape 
from a requirements perspective. The key requirements were drawn from the sources 
shown in Appendix B. 

4. Capability Requirements 

Table 1 depicts the Capability Portfolio Analysis as a function of Cost, Schedule, 
and Performance. 


Table 1. Capability Portfolio Analysis as a Function of Cost, Schedule and Performance 


Portfolio Item 

Capability 

Development 

Capability 

Equipment 

Components 

Capability 

Interfaces 

Capability 

Installation 

Capability 

Life-Cycle 

Logistics 

Notional 

Baseline 

Current 

Available 

Capability 

Current 

Component 

Availability 

Current System 
or Equipment 
Interoperability 

Historical 

Installation 

Cost and 
Schedule for 
similar 
Capability 

Historical 
administration, 
training, parts 
kits, 

publications, 
design changes, 
etc. for similar 
Capability 

Cost as a 
Function of 
Increase, 
Decrease, or 
Change to any 
Portion of the 
Program 

Changes to 
application of 
the Capability 
(all changes 
equate to cost 
increase and 
schedule impact 
+/-) 

Cost will be 
constant or an 
increase for 
systems if any 
components, 
software, etc. 
are changed. 

Cost increase 
may be offset 
by reduction in 
components. 
May also affect 

Cost may be 
same as notional 
or will be + for 
integration with 
any new 
components. 
Equates to 
information 
distribution, 
whether internal 
to the equipment 
or as external 

Cost may be 
same as 
notional cost, 
an increase, or 
a decrease 
depending upon 
a number of 
factors, 
including 
location, 
difficulty of 
installation or 

Only historical 
baseline costs 
will remain the 
same. Any 
change, 
regardless of 
increase or 
decrease in 
capability will 
equate to a 
positive cost 
over baseline. 
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Portfolio Item 

Capability 

Development 

Capability 

Equipment 

Components 

Capability 

Interfaces 

Capability 

Installation 

Capability 

Life-Cycle 

Logistics 



schedule and/or 
performance. 

interfaces. 

delivery, and 
differentials to 
security, 
environment 
and others. 


Schedule as a 
Function of 
Weighted Tasks 
and Earned 
Value Metrics 

Any changes to 
baseline 
capability will 
have an impact 
to schedule, but 
established 
delivery date 
must be the 
goal. 

Whether COTS 
or GOTS 
based, 
component 
availability that 
satisfies the 
capability 
affect schedule 
+/-. 

Level of 
integration of 
components of 
the capability 
affects tasking 
and Earned 

Value within the 
schedule +/-. 

Degree of 
difficulty, 
location, etc. 
will affect 
schedule +/-. 

Procurement 

and 

Documentation 
approval may or 
may not affect 
overall schedule 
+/-. 

Performance as 
a Function of 
Technology 
Readiness 

Levels (TRL) 
(1-9) and 
Manufacturing 
Readiness 

Levels (MRL) 
(1-10) 

Assume legacy 
capability as 
constant (TRL 

9 and MRL 10). 
Any change to 
capability will 
have 

commensurate 
impact on 
Technology or 
Manufacturing 
Readiness 
either +/-. 

Legacy 
components, 
whether COTS 
or GOTS are 
TRL/MRL 9/10 
respectively. 
Current state of 
portfolio 
components 
will impact 
cost, schedule, 
and 

performance. 

Higher 

TRL/MRL the 
less impact on 
cost and 
schedule, but 
may impact life 
cycle logistics, 
interfaces, and 
installation 
significantly. 

COTS vs. GOTS 
are significant 
factors for 
interfacing with 
other systems. 
Even with high 
TRL/MRL 
degree of 
difficulty may be 
deciding factor. 
COTS may be 
acceptable as 
standalone 
interim solution 
with a migration 
plan for later 
GOTS 
integration. 

Installation 

Phase assumes 
high 

MRL/TRL, but 
physical or 
electronic 
characteristics 
must be 
considered in 
performance as 
these may 
impact cost and 
schedule. 

Affected +/- 
depending on 
degree of 
acquisition 
management 
interfaces 
required and 
Performance 
Based Logistics 
level of effort 
for life cycle 
sustainment. 


The Cost of Capability Concept must also be considered, but may be viewed as a 
pre-expenditure, or constant plus a fixed fee for changes. The capability concept 
document is pre-portfolio selection, and assembly of capabilities as portfolios to meet the 
mission need should be based on this concept document. Inflation over time is a constant 
for any capability selection from the portfolios and is not considered a major factor in 
selection. 
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5. 


Functional Requirements 


The Requirements Matrix, Table 2, shows the payload of the SE requirements 
analysis. The requirements matrix provides the foundation for analyzing the products 
vis-a-vis the requirements. The requirements are specified using performance 
characteristics which are decomposed to include functional aspects, system aspects, and 
identification of tests that are used to ensure the requirements are satisfied. 


Table 2. Requirements Matrix 


Category 

Classes 

Measures 

Metrics 

Group By 

Notes 

Operations 

Mission 

Effectiveness 

Speed of 
command 

Time 

Capability, 

Service 

Fact 



System 

Availability 

Ao 

System 

Fact 



Quality 

Morbidity 


Facts, 

Score 



User Defined 

[numeric value] 


Score 



Public Service 

0/1 (Binary) 


Score 



Transformational 

Impact 

0/1 (Binary) 


Score 



Situational 

Awareness 

#of Tracks 


Fact 



Interoperability 

Military 

Standard 


Score 

Category 

Classes 

Measures 

Metrics 

Group By 

Notes 

Logistics 

Footprint 

Lay down Effects 

Space, 

Inventory 

Quantity, 

Training 

Material, 

Non-Material 

Facts, 

Scores 


Acquisition Risk 

Item Value 

Unit Cost 


Fact 


Supplier 

Leverage 

Number of 
Suppliers 

#Qualified 

Suppliers 


Fact 


Total Cost of 
Ownership 

Life cycle Cost 

Complete 

System Costs 


Fact 

Category 

Classes 

Measures 

Metrics 

Group By 

Notes 

Strategic 

Alignment 

Business Model 

Organizational Fit 

0/1 (Binary) 

Strategies, 
Term (short, 
long) 

Score 


Organizational 

Impact 

Training 

Cost 


Fact 



Personnel 

#People, #Man 
Hours 


Facts 


Business Process 
Improvement 

Scope 

ROI/EMV** 


Fact 
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Category 

Classes 

Measures 

Metrics 

Group By 

Notes 

Cost/Benefit 

Dollar Value 

Return On 
Investment 

Cost, Payback, 
Payback Period 

Phase 

(RDT&E, 

Production, 

O&S) 

Fact 


EMV 

Return On 
Investment 

Cost, EMV, 

EMV Payback 
Period 


Fact 

Category 

Classes 

Measures 

Metrics 

Group By 

Notes 

Execution 

Performance 

Schedule 

EVM 

Schedule 

Value 

Program, 

Service, 

Phase 

Fact 


Cost 

EVM 

Cost Value 


Fact 

Category 

Classes 

Measures 

Metrics 

Group By 

Notes 

Risk 

Probability 

Rating 

1 through 5 

Ratings, 

Percentage 

Program, 

Service, 

Phase 

Score 


Consequence 


1 through 5 

Ratings, 

Percentage 


Score 

Category 

Classes 

Measures 

Metrics 

Group By 

Notes 

Organizational 

Impact 

Training 

People Affected 

Count 

Program, 
Service, Phase 

Fact 


Personnel 

People Affected 

Count 


Fact 


Quality of Life 

Surveys 

Survey Metrics 


Score 


** The intent of EMV is to create a single metrics equivalent to dollars in commercial PA. 


B. ANALYSIS 

1. Portfolio Capability Evaluation 

From the DoD acquisition perspective, capabilities to meet identified 
requirements that satisfy mission gaps in execution of military strategies are tied to cost, 
schedule, and performance. In turn then, it is logical to view portfolios in a manner that 
optimizes these values. Let us consider that, for an optimum identified capability 
requirement, the following is true, regardless of whether it is from existing legacy, 
evolutionary, or new development (Figure 4): 
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operating Capability+ 

Capability = ^ Capability 

concept Capability— 

Capability Capability Capability Capability Capability Capability 

Concept Development Equipment Interfaces Installation Logistics 

Figure 4. Capabilities Equation 

The summation of sub-folders in a notional capability portfolio will result in a 
final capability that matches the DoD mission gap requirement, is better than the 
requirement, or is somewhat less than the requirement, but is good enough to serve as an 
interim solution (a lower threshold must be established to know when a sub-folder must 
be discarded as not useful). In turn, each of the pieces of this notional capability (let us 
call them $K or notional constant dollars for the baseline) portfolio may be further 
decomposed such that (Figure 5): 


CC + 

+*> + $K ce + '°'- i + $K C +* A +« cl /' +A + 

*K ">9 A 

Where: $K cc 

Is cost of capability concept 


$K . 
cd 

Is cost of capability development 


■£/+ 

* 

o 

CD 

Is cost of capability equipment/components 


cif 

Is cost of capability interfaces 


$K 

cm 

Is cost of capability installation 


$K log 

Is cost lifecycle logistics forthe capability including contract, initial administration, training (T), 
publications (P), spare parts kits (Spk), design changes (Sys) forthe installation, software 
packages, etc. 

TA + PA+ J$?kA + SpsA = log A 


Figure 5. Summation of Capabilities 


As shown above, changes (or the +/- deltas) to the notional baseline capability 
result in increases/decreases for cost, schedule and/or performance. At the enterprise 
level, then, these parameters may be used to graphically show advantages and 
disadvantages for various options within each of the portfolio sub-folders (we will 
examine the individual parameters graphically later on). This same rationale may be used 
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for evaluating schedule and performance. When numerical values are assigned, 
portfolios of capabilities and their components may be evaluated to select those most 
favorable within cost, schedule and performance desired to meet the capability gap. Note 
that risk is not an issue in capability parameter selection at this point. Risk is an integral 
part of sub-folder or technology evaluation that may make a specific capability option 
within the portfolio infeasible. Risk may be evaluated at each enterprise level calculation 
and at each subsequent parent-child decomposition, such that risk is always a 
consideration throughout the selection process. 

Taken in aggregate, a graphical representation of notional capability fielding may 
be presented, as depicted in Figure 6. Once analyses of different aspects of the portfolio 
(the parametric deltas from notional baseline) are examined, the graphical representation 
of the actual effort may be very different. Minimum and maximum arbitrary thresholds 
will determine whether the portfolio sub-folder capability option is able to meet cost, 
schedule, and performance within the timeframe desired by the tasking activity. If yes, 
then the portfolio sub-folder should be examined further. If no, then the capability option 
should be tabled for future consideration, or as a “fallback” option. 



Figure 6. Graphical Representation of Notional Capability Fielding 
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In measuring the parametric deltas/differences for a cost valuation, consider a 
matrix similar to Figure 7, where notional cost is shown as a function of historical 


spending rates for capability development and implementation: 


Total percent 
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Figure 7. Notional Cost as a Function of Historical Spending Rates 


Adding or subtracting different parameters desired from the capability portfolio 
sub-folders gives realistic notional thresholds for where expected costs will fall 


throughout the capability development and fielding, as depicted in Figure 8. 



Figure 8. Realistic Notional Cost Thresholds 
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Evaluation of the portfolio must also consider the desired timeframe for fielding 
the capability. If technologies or equipment within the selected sub-folders have low 
probability of reaching the fielding date, they must be shelved until they are mature 
enough for consideration. Schedules for the selected comparison parameters must use 
arbitrary weighted earned values. Depending upon the capability requirement, there are 
several types of scheduling software products that may be used. Regardless of the 
scheduling tool, as long as it is consistent, a reasonable comparison may be made 
between portfolio sub-folders that meet notional Earned Value Management System and 
Integrated Defense Acquisition, Technology, and Logistics management framework 
milestone and alternative decision point requirements. 
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lifecycle logistics for the capability must keep pace with development, contracting, 
and Testing 


Figure 9. Simple Capability Portfolio 


Very simple capability portfolios may only require something as elementary as 
Figure 9 above to select the best options within the sub-folders. Other complex 
capabilities may require in-depth project breakdown, and individual parameter 
comparisons within established thresholds, as shown in Figure 10. Sub-folders may also 
have spin-out increments to fall back on legacy technologies, should risk mitigation be 
insufficient to continue a specific variable. 
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Figure 10. Complex Capability Portfolio 


Evaluation of possible capability variances may be set down in simplistic 


spreadsheets as depicted in Figure 11 below: 


SCHEDULE 

as a function 
ofweighted 
Earned Value 
Tasking 

Q1 

Q2 

Q3 

Q4 

Q5 

Q6 

Q7 

QB 

Q9 

Q10 
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JCIDS 

Capability 
Development 
Capability 
Equipment & 
Components 
Capability 
Interfaces 
Capability 
Installation 
Capability 
Lifecycle 
Logistics 


MS A 




MS B 




MS C 

LRJP 

FRP 




Option A 



Option B 









spinout 
option A 


spinout 
option B 











option A 



option B 



























PBL 



Figure 11. Evaluation of Possible Variances Depicted in Spreadsheet 


When overlaid upon the notional schedule with scheduling thresholds, as shown 
in Figure 12, one can readily see where spinouts might be implemented across the 
capability fielding process. This process is especially useful in “design to price” or 
“turnkey” delivery of capabilities. 
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Figure 12. Notional Schedule with Scheduling Thresholds 


Although the capability performance may be measured by several criteria, it may 
be best measured via established criteria, as set forth in a Technology Readiness 
Assessment and/or Manufacturing Readiness Assessment. These criteria are well- 
recognized throughout the DoD and other government activities. Because of the 
expandable nature of the requirements for each level of performance, each of the 
parameters for evaluating a sub-folder within the portfolio may be tailored to fit the 
criteria of technology and manufacturing readiness. When compared with one another, 
using the same criteria, selection of the most promising sub-folders for the capability can 
be accomplished. 

Because sub-folders consider existing, evolutionary, and new developments, use 
of this method allows the evaluator to be able to inject capabilities into the performance 
versus time chart at the current level and select those that may be more mature and, 
therefore, have the best chance of success, all other weighted factors being near equal. 
The evaluator must be attuned to the pitfalls of selecting mature technologies, even 
though they meet a current need, which cannot evolve and remain interoperable with 
other capabilities in the battle space in the out years. A simplistic summary of 
technology and manufacturing readiness and assessment criteria is shown in Figure 13. 
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Figure 13. Summary of Technology & Manufacturing Readiness 

and Assessment Criteria 


In evolutionary and developmental sub-folders, it must be noted that the notional 
performance of the capability must be at a TRL of 6 by Milestone B, as set forth in Title 
10 (see Figure 14). 



Figure 14. Capability at Technology Readiness Level 6 by Milestone B 
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As with scheduling, a simplistic spreadsheet, as depicted in Figure 15 below, may 
be completed for the notional capability and then parameters from the sub-folders 
overlaid to match projected performance growth over the notional timeline for the 
required fielding date of the capability. Again, there are many software tools, both 


simple and complicated, that may be used to perform this function. 
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Figure 15. Simplistic Spreadsheet Showing Notional Performance Capability 


Notional capability parameters may vary, depending upon the type of capability 
being fielded, and the readiness level of the parameters may vary widely, depending upon 
the maturity of selected sub-folders to meet the capability. However, as seen in Figure 15 
above, to meet fielding criteria there are accepted readiness levels that are expected at 
certain proposed decision points and milestones for the capability. 

2. Threat Environments (Based on the 2008 National Defense Strategy 
(NDS) 

a. Threat Environment Current 

Today’s current environmental threat is: Global struggle against violent 
extremist ideology seeking to overturn the international state system, 
asymmetrical/irregular warfare. 
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b. Threat Environment Future 

Irregular challenges, rogue states quest for nuclear weapons (Iran and 
North Korea), rising military power of other states (China, potentially Russia). 

c. Objectives 

The objectives are as follows: 

• Defend the Homeland 

• Win the Long War 

• Promote Security 

• Deter Conflict 

• Win our Nation’s Wars 

d. Achieving the Objectives 

Objectives are met by shaping the choices of key states, preventing 
adversaries from acquiring or using Weapons of Mass Destruction, strengthening and 
expanding alliances and partnerships, securing U.S. strategic access and retaining 
freedom of action, and integrating and unifying our efforts. 

e. Risks 

Since it is not possible to fund every project, ultimately choices must be 
made. Here risk is defined in terms of the potential for damage to national security 
combined with the probability of occurrence and a measurement of the consequences 
should the underlying risk remain unaddressed. The first risk is that partner contributions 
to future coalition operations will vary in size, composition, competence, and capability. 
Second, the risk strategy must account for four dimensions of risk: 

Operational Risks - Those associated with the current force executing the 
strategy successfully within acceptable human, material, financial, and strategic costs. 

Future Challenges Risks - Those associated with the Department’s 
capacity to execute future missions successfully against an array of prospective future 
challengers. 
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Force Management Risks - Those associated with managing military 
forces fulfilling the objectives described in the NDS. The primary concern here is 
recruiting, retaining, training, and equipping a force and sustaining its readiness. 

Institutional Risk - Those associated with the capacity of new command, 
management, and business practices. 

3. Gartner Group’s COTS PA Product Landscape 

Applications for project, portfolio and resource management boost team 
performance and enable IT management and others to access real-time data via 
dashboards for prioritization and quick decision making. As clients look to better manage 
IT portfolios, vendors are marketing increasingly integrated Project Portfolio 
Management (PPM) suites for strategically aligning and planning IT projects, and for 
controlling application portfolio costs. The dynamic PPM market is poised for an annual 
growth rate of 12% to 20% through year end 2012, as users seek to tie strategies more 
effectively to planning and control—including IT strategy. As shown in Figure 16, the 
top four PPM vendors worldwide in 2007, in terms of total software revenue, were (in 
order) Microsoft, Primavera Systems, CA and Planview. These top four vendors held 
46% of the total worldwide PPM market share in 2007, with revenue of more than $512 
million (copyright Gartner Group, 2008). 
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Figure 16. Top Four PPM Vendors Worldwide in 2007 


33 




Our research team will research the top four PPM solutions and a employ a 
vertically focused Risk Management solution to evaluate which is the best or if none are 
capable of providing an all-in-one solution. Our analysis will not only require program 
management but critical portfolio, decision, analysis and the ability to provide DoD 
analysis capability. 


a. COTS PPM Suite#l: Microsoft Project Portfolio Management 

Program 


Microsoft provides an interesting approach to PPM, with use of its Office 
Project Portfolio Server (OPPS) 2007, which assists organizations in realizing its 
potential by identifying, selecting, managing, and delivering portfolios that align with 
strategic priorities. 


CREATE 


SELECT 


PLAN 


MANAGE 


Complete 

Request 



Figure 17. OPPS 2007 Workflow Design 


Figure 17 above includes an intuitive workflow design as part of a 
tracking tool that is designed for project/program managers to define control points to 
multiple workflows. The control points govern life cycle phases used as a co mm on 
denominator to aggregate and report on projects across various fields. The beauty of this 
deliverable is that it receives managerial signature before moving forward in the life 
cycle phases. This audit functionality makes stakeholders aware and accountable as 
projects move from ideas to consideration to implementation. 
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Figure 18 below describes analysis techniques that create rapid 
assessments of the impact on the portfolio’s business value and effectively communicates 
the tradeoffs, including the listed projects within a portfolio. 
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Figure 18. Analysis Techniques for Rapid Assessment of Impact 
on the Portfolio’s Business Value 


Figure 19 below is an example of a tracking tool from OPPS, which is 
designed to measure and track projects, programs, and applications throughout their life 
cycle. This provides the visibility to proactively identify potential issues, promote 
decision making, and help ensure that the organization portfolios deliver maximum 
business value. 
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Figure 19. Sample OPPS Tool for Measuring and Tracking Projects/Programs/Applications 

Throughout Their Life Cycle 

OPPS is designed to support open architecture, enabling integration with 
Microsoft Office Project Server 2007 and other PPM COTS applications to provide 
organizations with an end-to-end project portfolio management solution. The bi¬ 
directional gateway enables administrators to link multiple Microsoft Office Project 
Servers to OPPS 2007, providing managers with a consolidated view of all projects 
within the organization. 

In summary, Microsoft optimizes the use of OPPS and selects portfolios 
that best align with an organization’s strategic priorities. The interface provided is user- 
friendly, and embeds best practice methodologies and analytical techniques to ensure 
selection of the right investments for business solutions. Microsoft shares its Corporate 
Project Solutions, allowing organizations to: 

• Employ sophisticated optimization algorithms to select the optimal 
portfolio under varying budget constraints, such as costs and Full Time 
Equivalents (FTEs). 
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• Identify and break the constraints prohibiting the portfolio from reaching 

the Efficient Frontier. 

• Automatically calculate the portfolio’s alignment with an organization’s 

business strategy. 

• Enforce a rational, rather than emotional, approach to selecting portfolios. 

(Microsoft PPM, 2009) provides a link to the Microsoft PPM Web site on 

the Internet. 

b. COTS PPM Suite #2: Primavera Systems Project Portfolio 

Management 

Customers use Primavera as a project, resource and portfolio management 
solution to propose, prioritize, and select strategic investments and then to plan, manage 
and control the entire portfolio of projects through to successful completion. Primavera 
solutions are industry-specific and role-based, with a design powered to support global 
enterprises. Primavera PPM product can be used to manage any type of portfolio, 
including: 

• IT project requests 

• Applications for rationalization 

• New product development 

• Capital programs 

As depicted in Figure 20 and Figure 21, utilizing Microsoft Internet 
Explorer, Primavera’s design supports everything from data entry forms for details about 
a single investment, to scorecards for evaluating a set of investments in a portfolio, to 
investor maps for reviewing and analyzing a portfolio of investments, or even evaluating 
a portfolio of portfolios. 
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Figure 20. Sample Primavera Project Portfolio Management Tool 


The key aspect of Primavera PPM is that it is specifically designed for 
proposing, planning, and controlling portfolios of investments in a collaborative way, 
following an objective and transparent process. These results propose enormous gains in 
efficiencies and improvements in business performance. 



> PRIMAVERA ProSight Administrator, System | Sunday, Dec 16, 2007 




^ 5 ] Item: Accounting Database 1/39 


-1 -| -1- 

© Trusted sites 

Start] J yj ^ ^ f -% -A. 

” | ri*!l ”1 Prima ProSight Demo.doc... | * ) Prima ProSight Demo 

| Primavera ProSight - Sco... | 

« g£)5 : ' : ' 2:51 PM 


Figure 21. Primavera PPM Sample Application Analysis 
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In summary, Primavera PPM provides portfolio management software 
solutions, with similar Microsoft flexibility and infrastructure for enterprise portfolio 
management. The unique functionality, real-world enterprise scalability, and unlimited 
configurability make Primavera PPM a viable solution for managing just about any type 
of portfolio from ideas through execution. It also is designed with powerful security 
features for clean user interface and collaboration among all organizational stakeholders. 
(Primavera PPM, 2009) provides a link to the Primavera PPM Web site on the Internet. 

c. COTS PPM Suite #3: Planview Portfolio Management Program 

Planview has designed tools to identify project requirements, manage the 
scope of work, and minimize change. These tools provide more visibility into how a 
project can stay on track and deliver real business value. The Planview PPM COTS 
application was designed to provide capabilities to ensure: 

• Optimization of resources enterprise-wide 

• Integrated decision making 

• Ability to view project performance and perform trend analysis 

• Mitigation of risks and management of changes 

As depicted in Figure 22 below, Planview PPM COTS application design 
assists in balancing portfolios, supporting management decisions for the proper allocation 
of work to the most appropriate personnel. By managing work with this portfolio, one 
can deliver the highest business value to projects across an organization. 
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Figure 22. Planview PPM Application Development Performance Portfolio 


In addition, project budgets must be forecasted, baselined, and managed 
throughout the project life cycle. With Planview Enterprise, IT financial managers, the 
PMO and project managers can collaborate to better forecast costs and monitor spending 
through actual resource assignments and reported time. The key is the establishment of 
integration with existing financial and accounting systems with Microsoft, as depicted in 
Figure 23 below. 
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Figure 23. Planview Enterprise Tool for Forecasting Costs and 

Monitoring Spending 


Plainview is designed to help technology organizations deliver value 
through a comprehensive, systematic approach to: 

• Measuring and analyzing current performance, 

• Comparing actual results with clearly defined business 
requirements, 

• Analyzing and documenting decisions, and 

• Efficiently executing change initiatives. 

(Plainview PPM, 2008) provides a link to the Planview PPM Web site on 

the Internet. 
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d. COTS PPM Suite#4: CA Project Portfolio Management 

The CA PPM approach facilitates collaborative projects between 
organizations and CA, based on repeatable best practices guided by forecasted 
implementation from leadership. 

During the design and development process, CA and CA strategic partners 
work together with organizations to: 

• Assess and prioritize requirements, 

• Create a requirements specification and project plan, 

• Architect a process and technical design and implementation blueprint 
to improve maturity, 

• Install, configure, integrate and test a solution “as built” for 
environments and improve processes, and 

• Deliver quality results quickly in 30 days or less in order to achieve 
rapid time-to-value and improved business alignment. 

In addition, like any project, CA PPM focuses on strong sponsorship from 
senior management, along with representation from the various stakeholders within the 
organization who understand the drive needed to help cultivate the business benefits of 
enhanced change. CA uses a deployment methodology which is certified to align 
organization PPM capabilities and designs for strategic service life cycle management 
goals. Below are some CA focal points that are designed for PPM solutions: 

a. Achieve Rapid Time-to-Value through repeatable scoping models 
and gap analysis that identify role-based business objectives, requirements and 
configuration designs, plus deployment methodologies that apply and adapt the pre¬ 
defined configurations of CA Services Accelerators to deliver a custom CA PPM 
solution. 

b. Implement Multi-phased Deployments with a goal of 60-90 day 
increments so you can continually deliver IT service value and improve the maturity of 
your CA PPM solution without overwhelming your organization with change and 
excessive adoption challenges. 
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c. Stabilize your Solution at the End of Deployment phases to 
maximize your current investment. Consultants tune your CA PPM solution, provide 
administrative knowledge transfer to your IT staff and solution overviews for your end 
users, assist with production rollout and make recommendations that can speed adoption 
or further develop your CA PPM solution maturity and business value. 

d. With On-boarding Services from CA Education, you can achieve 
expected productivity gains and increase adoption rates. A robust selection of prepared 
courses delivered in a variety of formats and options, custom-designed courses, and the 
CA Productivity Accelerator (a context-sensitive, customizable training solution) provide 
focused, ongoing education support to speed adoption of your CA PPM solution. 

The touted benefits of CA PPM are designed to provide: 

• Lower risk from experience and best practices. 

• Faster results through incremental deployment methodologies, self- 
service support knowledge and in-context and online training. 

• Improved outcomes with thought leadership and proven experience. 

• Knowledgeable administrators and users from deployment on- 
boarding, continuing education and support guidance. 

(CA PPM, 2008) provides a link to the CA PPM Web site on the Internet. 

4. Forrester Research COTS PA Product Landscape 

The Forrester Wave™ Project Portfolio Management Tools, Q4 2007 report 
established CA as the PPM leader in 2007, according to an article published 18 
December 2007, by Mr. Lewis Cardin et al. (p. 8). Forrester evaluated fourteen leading 
PPM vendors across ninety-five criteria and found that CA and Planview established 
PPM leadership within the field thanks to their wide choice of mature features and 
functions. Forrester’s COTS PA product research uncovered a market environment in 
which: 

• CA, Planview, HP, Primavera, and IBM lead the pack 

• Compuware, Oracle, Serena, and Microsoft offer competitive options 

• SAP and Daptiv lack the expected full suite of out-of-the-box offerings 
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This evaluation of the COTS PPM Suite market is intended to be a starting point 
only. Readers are encouraged to view detailed product evaluations and adapt the criteria 
weightings to suit their individual needs through the Forrester™ Wave Excel-based 
vendor comparison tool (see Figure 24). 


Risky Strong 

Bets cointei>ders Performers Leaders 



Figure 24. Forrester™ Wave Portfolio Management Tools, Q4’ 07 
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As the front runner, CA is not alone and, by far, has not cornered the market in 
the PPM arena. However, as shown above, growth is taken seriously by all players to 
become the best in an economy that could use leadership. Below are company profiles 
that contributed to their PPM leadership roles: 

• Microsoft moves into the leader zone. Microsoft has had some mountains to 
climb with its integration of UMT (a premier portfolio management 
application) and getting off its thick client workstation solution for project 
management. Microsoft has succeeded in gluing these projects and portfolio 
management solutions together, along with the advent of its server 
technology. While it’s reporting capabilities continue to excel (no pun 
intended), its customers will benefit from further development in its 
methodology offering and increased financial management capability, 
particularly in the area of chargeback (Microsoft Advances As A Leader In 
The PPM Market (Cardin, Lewis, Forrester Wave, 2007). 

• Primavera continues to keep a stronghold with its demand management, 
workflow, and methodology software; it must develop more depth in its 
Application Portfolio Management and infrastructure portfolio management 
offerings to be as equally attractive to its IT customers as it is with rest of the 
enterprise (Primavera Is A Leader In The Large Enterprise PPM Market, 
(Cardin, L., Cullen, A., & Cecere, M., 2007). 

• Planview continues to be in front in the specific area of portfolio management 
and is neck and neck with Primavera on project and portfolio management 
methodology. Planview still has some work to do with its integrated IT 
management offering (Planview Is An Undeniable Leader In The PPM 
Market, (Cardin, L. Cullen, A., & Cecere, M., 2007). 

• CA’s continued strength in reporting and its focus on enterprise IT 
management makes PPM an integral part of overall customer IT management 
(CA Leads in Many Offerings in the PPM Market (Cardin, L., Cullen, A., & 
Cecere, M., 2007). 
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5. 


Real Options Valuation’s Risk Simulator 


Real Options Valuation’s Risk Simulator software package will help to identify, 
quantify, and evaluate risk in projects and decisions. Risk Simulator, a powerful Excel 
add-in application, is used for applying simulation, forecasting, statistical analysis, and 
optimization in existing Excel spreadsheet models. It contains four different modules: 
Monte Carlo Simulation, Optimization, Statistical and Analytical tools, Time Series and 
Cross-Sectional forecasting. Risk simulator is also integrated with the Real Options 
Super Lattice Solver software, for solving strategic real options, financial options, and 
employee stock options. 

Figures 25 through 29 depict sample optimization models which were run with 
Risk Simulator, both before and after optimization. Figure 25 is a discrete model that 
uses a static optimization which is run on static models, no assumption inputs, without 
simulations. This type of optimization is usually run to determine the initial optimal 
portfolio before more advanced optimizations are applied. 


5 Discrete Model 1 - Microsoft Excel 
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ENPV is the expected NPV of each credit line or project, while Cost can be the total cost of 
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administration as well as required capital holdings to cover the credit line, and Risk is the 
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Coefficient of Variation of the credit line's ENPV. 
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Figure 25. Sample Optimization Model Run with Risk Simulator (Discrete Model) 


Note that the selection criteria are all “ones.” The decision criteria for this model 
is to select six of the best projects based on risk and cost, with a cost limit of less than 
$5000. Figure 26 below is the same model after being run through 250 iterations and 
evaluating the risk/cost factors to make the best decision. 
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Discrete Model IRun 250 Iterations - Microsoft Excel 
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ENPV is the expected NPV of each credit line or project, while Cost can be the total cost of 
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Figure 26. Sample Optimization Model Run with Risk Simulator (After Run Through 250 

Iterations) 

Figure 27 below is a sample of a Stochastic Model, which can be run either by 
dynamic optimization or by stochastic optimization. Dynamic optimization is a first-run 
simulation. The results of the simulation are applied to the model, and then an 
optimization is applied to the simulated values. 
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Specifications of the optimization model: 
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Objective: 


Maximize Return to Risk Ratio (B13) 
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Decision Variables: 

Allocation Weights (D6.D10) 














Restrictions 
on Decision 
Variables: 


Minimum and Maximum Required (E6:F10) 
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Constraints: 



Portfolio Total Allocation Weights 100% (D12 

is set to 100%) 
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Additional specifications: 
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1. One can always maximize portfolio total returns or 

minimize 

the portfolio total risk. 
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2. Incorporate Monte Carlo simulation in the model by simulating the returns and 

volatility of each asset class 
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and apply Simulation-Optimization techniques. 
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3. The portfoli 

o can be optimized as is without simulation using Static Optimization techniques. 











Figure 27. Sample Optimization Model Run with Risk Simulator 

(Stochastic Model) 
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Figure 28 below is the same model run through the dynamic optimization with 
1000 simulation trials. 
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Specifications of the optimization model: 
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Objective: 
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Decision Variables: 

Allocation Weights (D6.D10) 
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Restrictions on Decision Vari Minimum and Maximum Reauired (E6:F10) 
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Constraints: 


Portfolio Total Allocation Weights 100% (D12 is set to 100%) 
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1. One can always maximize portfolio total returns or minimize the portfolio total risk. 
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2. Incorporate Monte Carlo simulation in the model by simulating the returns and volatility of each asset class 
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and apply Simulation-Optimization techniques. 











27 

3. The portfolio can be optimized as is without simulation using Static Optimization techniques. 












Figure 28. Sample Optimization Model Run with Risk Simulator 
(Dynamic Optimization with 1000 Simulation Trials) 


Note that the rankings have changes based on the simulations comparing the input 
assumptions (green boxes) and the decision criteria (blue box). The light blue boxes are 
the objective, risk return ratio (B13), and the constraint (100%, (D12)). 

Stochastic optimization is similar to dynamic optimization, except that the process 
is repeated several times. The final decision variables will each have their own forecast 
chart indicating their optimal range. Figure 29 below is the same model run through the 
stochastic optimization with 1000 simulation trials and 20 optimization runs. 
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Stochastic Model 1 Run - Microsoft Excel 
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Specifications of the optimization model: 
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Objective: 


Maximize Return to Risk Ratio (B13) 
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Decision Variables: 


Allocation Weights (D6.D10) 
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Restrictions on Decision Var Minimum and Maximum Required (E6:F10) 
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Constraints: 

Portfolio Total Allocation Weights 100% (D12 is set to 100%) 
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Additional specifications: 
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1. One can always maximize portfolio total returns or minimize the portfolio total risk. 
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2. Incorporate Monte Carlo simulation in the model by simulating the returns and volatility of each asset class 
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and apply Simulation-Optimization techniques. 












27 

3. The portfolio can be optimized as is without simulation using Static Optimization techniques. 



































Figure 29. Sample Optimization Model Run with Risk Simulator (Stochastic Model with 

1000 Simulation Trials and 20 Optimization Runs) 


Again, note the changes in the rankings and the objective. 


Figure 30 below depicts the statistical results of the optimization for Program 3, 
with 20 data points. 


S3 Program 3 

84 NumtjerofDafapOiiifs 

85 Wean 

86 Median 

87 Standard Deviation 

88 Variance 

89 4'/erage Deviation 

90 Maximum 

91 Minimum 

92 Range 

93 Skewness 

94 Kurtosis 

95 25% Perrenffie 

96 75% Pencenffj'e 

97 Ernor Precision at 95% 

__ 



Figure 30. Statistical Results of Optimization for Program 3 (with 20 Data Points) 
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The Risk Simulation software provides a risk/decision analysis to help any 
business make better decisions, both now and in the future. 

Efficient Frontier (EF) analysis, shown in Figure 31 below, is the process of re¬ 
running multiple optimizations with different constraints and each optimal portfolio is a 
point on the frontier. To run the analysis, click on the Constraints icon or in Risk 
Simulator, select Optimization and Constraints. Then add as many changing constraints 
as desired to generate multiple EFs. 



S3 



Optimal values have been found. Do you wish to replace the enListirwj decision variables with the opti mi zed values or 
revert to the original inputs? 


| Create Report | | Cancel 


Figure 31. Efficient Frontier 


6. Palisade @RISK (http://www.palisade.com/) 

Palisade @RISK performs risk analysis using Monte Carlo simulation to show 
you many possible outcomes using Microsoft Excel spreadsheets, and tells you how 
likely outcomes are to occur. You can then decide which risks are worth taking and 
which ones to avoid, allowing for improved decision making under uncertainty. @RISK 
uses simulation to answer questions like: 

a. “What is the probability of profit exceeding $ 10 million?” 

b. “What are the chances of losing money on this venture?” 
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c. 


“What is the probability that the project will be delivered within budget?” 


d. “How much contingency (management reserve) should be included?” 

@RISK is an add-in to Microsoft Excel, integrating completely with the 
spreadsheet. All @RISK functions are Excel functions, and behave exactly the same as 
native Excel functions. @RISK windows are all linked directly to cells in your 
spreadsheet, so changes in one place are carried over to the other. @RISK graphs point to 
their cells via callout windows. Figure 32 depicts an @RISK probability distribution for a 
sample project. 



Figure 32. @RISK Probability Distribution 


7. Shortcomings of COTS Products 

The drive to incorporate COTS software and hardware is often based on 
incomplete or inaccurate information. Clearly identified requirements that cannot 
incorporate COTS software and hardware need additional research, cost, and 
development within their product design. However, using PPM COTS products does 
offer many advantages and disadvantages. For instance, many of these shortcomings are 
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described by engineers who are reluctant to move to COTS, if for no other reason than a 
general resistance to change. Many engineers display the attitude “if we did not make it; 
the product design is no good.” It is basically that type of mentality that promotes a 
general distrust of COTS products. 

a. Advantages of COTS 

The advantages associated with the use of COTS are: 

• Immediately available and shorter development schedule 

• Reduced cost 

• Increased portability 

• Improved Quality (resulting from more efficient testing) 

b. Disadvantages of COTS 

The disadvantages associated with the use of COTS are: 

• Hard to meet special requirements 

• Continual investment in COTS product 

• Bad interoperability 

Although it is generally a good idea not to accept a product without verifying its 
capabilities, it is also not a good idea to reject a product on general principle. Below are 
the shortcomings of discussed COT products and assumptions about their acceptable 
levels of requirements that need to be or have been adjusted for product/solution. 

a. Microsoft’s design lacked Portfolio management requirements to 
gather accurate data, which led to the acquisition of UMT. This compensated for the 
needed solution and supported the project management software deployed in late 2007. 

b. Primavera has mastered the art of determining trends in the PPM 
market, identifying the PPM needs of its target markets, and acquiring solutions that fit 
the vendor’s ever changing and evolving products. The design has shortcomings that are 
ideally suited to large enterprises with mature PPM processes; design is not suited for 
requirements of smaller-sized businesses. 
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c. Plainview’s design is the second strongest portfolio management 
solution respected on the market. 

d. CA is the leader and continuously moving up in its leadership position. 
Shortcomings are very limited to its continued investments in integrated IT management. 

C. MODEL DEVELOPMENT 


1. Model Data 


Using the basic data depicted in Figure 33 below, our research team attempted to 
develop a model in (1) Real Options Valuation’s Risk Simulator and (2) @RISK that 
would analyze the data and provide usable output as measures of EMV. We also 
attempted to model this data with the Program Management (PM) software but were 
unable to do so. The EMV measures for this model were developed by our team. 
However, in practice, the EMV measures should be developed by Budget, Strategic, and 
Acquisition professionals to ensure their accuracy. 


Program 

ENPV 

NPV 

Cost 

Strategy ” 
Ranking 

P-1 i 1 ita ry 

Score 

Tactical 

Score 

FTE 

Resoucres 



Program 1 


$501.60 

$102.30 

$443.50 

1.94 

2.51 

1.77 

3.91 



Program 2 


$726.30 

$121.30 

$353.30 

3.75 

3.84 

3.56 

2.73 



Program 3 


$932.40 

$354.20 

$925.90 

2.34 

3.12 

3.24 

5.24 



Program 4 


$620.50 

$265.30 

$512.10 

1.36 

5.14 

5.1 

3.41 



Program 5 


$420.80 

$271.20 

$524.50 

1.32 

3.22 

1.62 

3.29 



Program 6 


$2,087.50 

$345.90 

$1,368.70 

4.63 

6.37 

6.21 

4.56 



Program 7 


$3,264.90 

$594.10 

$1,602.10 

10.59 

3.69 

7.53 

4.23 



Program 3 


$2,563.40 

$421.50 

$2,133.40 

2.33 

2.64 

4.12 

5.61 



Program 9 


$5,463.10 

$1,032.20 

$2,712.30 

7.63 

6.94 

7.99 

4.68 



Program 10 


$3,243.90 

$775.40 

$2,502.30 

5.74 

5.53 

6.23 

6.43 



Program 11 


$125.90 

$32.30 

$28.60 

15.26 

9.34 

9.17 

5.47 



Program 12 


$235.10 

$85.60 

$109.50 

6.42 

4.25 

3.65 

3.91 



Program 13 


$4,163.50 

$325.10 

$3,054.20 

5.26 

3.41 

6.21 

5.23 



Program 14 


$4,463.70 

$594.10 

$1,673.80 

10.54 

8.46 

7.54 

4.23 



Program 15 


$587.60 

$421.50 

$409.00 

2.81 

2.74 

3.52 

5.34 



Program 16 


$1,698.40 

$431.90 

$653.30 

11.92 

3.45 

3.37 

4.26 



Program 17 


$758.40 

$134.20 

$435.50 

4.65 

4.52 

4.35 

6.32 



Program 18 


$1,243.30 

$301.50 

$331.50 

4.35 

6.13 

5.62 

3.57 



Program 19 


$389.90 

$26.10 

$159.00 

7.93 

7.65 

3.62 

4.93 



Program 20 


$3,364.70 

$1,257.60 

$1,453.70 

15.46 

9.43 

9.34 

12.64 














$21,952.20 

127.33 

112.93 

114.26 

100.04 













Maximize 

<=$10,500 

<= 100 



<= 30 



Objective: Maximize Total Portfolio Rot a ms times the Portfolio Comprehensive Score 

Decision Variables: Allocation or Go/No-Go Decision 

Restrictions on Decision Variables Binary decision variables (0 or 1) 

Constraints: Total Cost is less than $10,500 (in thousands or millions of dollars) 

Less than or equal to 10 projects selected in the entire portfolio 

Full-time Equivalence resources have to be less than 80 

Total Strategic Ranking for the entire portfolio must be less than 100 


Figure 33. Base Evaluation Model 
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Earned Net Present Value is an enhancement of the NPV that explicitly addresses 
uncertainty. NPV compares a single stream of cash flows in today’s dollars to the value 
of that same dollar in the future. Cost is the actual cost listed in the 2009 Defense 
Procurement Budget Request. Strategic Ranking, Military Score, and Tactical Score are 
EMV measures developed by evaluating the NDS and then scoring the programs based 
on how they meet the NDS. FTE resources equates to the amount of actual resources 
used as a percent. One hundred percent means that all resources are fully utilized all the 
time. The goal is to maximize the portfolio returns without exceeding an arbitrary budget 
of $10,500 while keeping the strategic ranking below 100 and the FTE below 80. 

In evaluating this model, we plan to verify that PA and Defense budget decision 
making can be improved using COTS software. 

2. Real Options Valuation’s Risk Simulator 

Figure 34 is the model developed using the Real Options Risk Simulator. The 
task was to run an optimization/simulation to determine which ten of the following 
twenty programs best meet the requirements outlined in the NDS. Only ten programs 
will go forward. These are real programs in the FY 2009 budget with the real costs 
included. The names of the real programs will be revealed later in the analysis section. 
The ranking and military/tactical scores, along with the FTE resources, are based on the 
NDS and the President’s goals are located: http://www.whitehouse.gov/agenda/defense/ . 

Because our thesis is based on PA, we will attempt to run this through PPM 
programs as well as risk simulators. For the purpose of this model, the military score, 
comprehensive score, and tactical score are all measures of EMV. 

After running the discrete (static) optimization on this model with the original 
budget of less than $10,500, and a strategy rank of less than 100, no more than ten 
programs, and FTE resources not to exceed 80. 
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Military Portfolio Optimization 





Project Name 

ENPV 

NPV 

Cost 

Strategy 

Ranking 

Return to 
Rank Ratio 

Profitability 

Index 

Selection 

Military Score 

Tactical 

Score 

FTE 

Resources 

Comprehensive 

Score 

Project 1 

$50160 

$102.30 

$443 50 

4.26 

117.75 

1.23 

1 

2.51 

1.77 

3.91 

1.40 

Project 2 

$726.30 

$121.30 

$358 80 

5.61 

129.47 

1.34 

1 

3.84 

3.56 

2.73 

1.77 

Project 3 

$932.40 

$35420 

$925.90 

2.34 

398.46 

1.38 

1 

3.12 

3.24 

5.24 

2.12 

Project 4 

$620.50 

$265.80 

$512.10 

3.21 

193.30 

1.52 

1 

5.14 

5.10 

3.41 

2.46 

Project 5 

$420 80 

$271.20 

$524.50 

1.32 

318.79 

1.52 

1 

3.22 

1.62 

3.29 

1.60 

Project 6 

$2.08750 

$345 90 

$1,868 70 

4 68 

446.05 

1.19 

1 

6.37 

6.21 

4.56 

298 

Project 7 

$3264.90 

$594.10 

$1,602.10 

10.59 

308.30 

1.37 

1 

8.69 

7.53 

4.23 

3.43 

Project8 

$2,563.40 

$421.50 

$2,133.40 

2.83 

905.80 

1.20 

1 

2.64 

4.12 

5.61 

2.31 

Project 9 

$5.46810 

$1,032.20 

$2,712.30 

7.68 

71199 

1.38 

1 

694 

7.99 

4.68 

3.46 

Project 10 

$3,248.90 

$775.40 

$2,502.80 

5.74 

566.01 

1.31 

1 

5.53 

6.23 

6.43 

3.16 

Project 11 

$125.90 

$32.80 

$28.60 

15.26 

825 

2.15 

1 

9.84 

9.17 

5.47 

4.13 

Project 12 

$235.10 

$85.60 

$109.50 

6.42 

36.62 

1.78 

1 

4.25 

3.65 

3.91 

2.00 

Project 13 

$4,163.50 

$325.10 

$3,05420 

5.26 

791.54 

1.11 

1 

3.41 

6.21 

5.23 

2.77 

Project 14 

$4,468.70 

$594 10 

$1,673.80 

10.54 

8.46 

7.54 

1 

8.46 

7.54 

4.23 

3.41 

Project 15 

$587.60 

$421.50 

$409.00 

2.81 

209.11 

2.03 

1 

2.74 

3.52 

5.34 

2.13 

Project 16 

$1.69840 

$431.90 

$65830 

11.92 

142.48 

1.66 

1 

8.45 

8.87 

4.26 

3.74 

Project 17 

$75840 

$134 20 

$485.50 

4 65 

163.10 

1.28 

1 

4.52 

4.35 

6.32 

2 60 

Project 18 

$1,248.30 

$30150 

$331.50 

4.85 

257.38 

1.91 

1 

6.13 

5 62 

3.57 

2 66 

Project 19 

$389.90 

$2610 

$15900 

7.98 

48.86 

1.16 

1 

7.65 

8.62 

4.98 

3.73 

Project 20 

$3.864 70 

$1,257.60 

$1,458.70 

15.46 

249.98 

1.86 

1 

9.48 

9.34 

12.64 

5.21 













Total 

$37,374.90 


$21,95220 

133.41 



1 20 | 

112.93 

114.26 

100.04 

57.08 

Profit/Rank 

$280.15 











Profit*Score 

$2,133,370.83 

Maximize 

<=$10500 

<=100 



x<=10 



<=80 



Figure 34. Real Programs in the FY09 Budget with Costs 

Figure 35 shows the model after the initial optimization run including the ten 
selected programs. The model was then run several more times, changing the constraints 
of budget, and number of programs allowed. This information was then used to update 
the data in the efficient frontier charts below. 


Military Portfolio Optimization 


Project Name 

ENPV 

NPV 

Cost 

Strategy 

Ranking 

Return to 
Rank Ratio 

Profitability 

Index 

Selection 

Military Score 

Tactical 

Score 

FTE 

Resources 

Comprehensive 

Score 

Project 1 

$501.60 

$102.30 

$443.50 

4.26 

117.75 

1.23 

0 

2.51 

1.77 

3.91 

1.40 

Project 2 

$726.30 

$121.30 

$358.80 

561 

129.47 

1.34 

0 

3.84 

3.56 

2.73 

1.77 

Project3 

$932.40 

$354.20 

$925.90 

2.34 

398.46 

1.38 

0 

3.12 

3.24 

5.24 

2.12 

Project 4 

$620.50 

$265.80 

$512.10 

3.21 

193.30 

1.52 

0 

514 

5.10 

3.41 

2.46 

Project 5 

$420.80 

$271.20 

$524.50 

1.32 

318.79 

1.52 

0 

3.22 

1.62 

3.29 

1.60 

Project 6 

$2,087.50 

$345.90 

$1,868.70 

4.68 

446.05 

1.19 

0 

6.37 

6.21 

4.58 

2.98 

Project 7 

$3,264.90 

$594.10 

$1,602.10 

10.59 

308.30 

1.37 

1 

8.69 

7.53 

4.23 

3.43 

Project 

$2,563.40 

$421.50 

$2,133.40 

2.63 

905.80 

1.20 

0 

2.64 

4.12 

5.61 

2.31 

Project 9 

$5,468.10 

$1,032.20 

$2,712.30 

7.68 

711.99 

1.38 

1 

6.94 

7.99 

4.68 

3.46 

Project 10 

$3,248.90 

$775,40 

$2,502.80 

5.74 

566.01 

1.31 

1 

5.53 

6.23 

6,43 

3.16 

Project 11 

$125.90 

$32.80 

$28.60 

15.26 

8.25 

2.15 

1 

9.84 

9.17 

5.47 

4.13 

Project 12 

$235.10 

$85.60 

$109.50 

6.42 

36.62 

1.78 

1 

4.25 

3.65 

3.91 

2.00 

Project 13 

$4,163.50 

$325.10 

$3,054.20 

5.26 

791.54 

1.11 

0 

3.41 

6.21 

5.23 

2.77 

Project 14 

$4,468.70 

$594.10 

$1,673.80 

10.54 

8.46 

7.54 

1 

8.46 

7.54 

4.23 

3.41 

Project 15 

$587.60 

$421.50 

$409.00 

2.61 

209.11 

2.03 

0 

2.74 

3.52 

5.34 

2.13 

Project 16 

$1,698.40 

$431.90 

$658.30 

11.92 

142.48 

1.66 

1 

8.45 

8.87 

4.26 

3.74 

Project 17 

$758.40 

$134.20 

$485.50 

4.65 

163.10 

1.28 

1 

4.52 

4.35 

6.32 

2.60 

Project 16 

$1,248.30 

$301.50 

$331.50 

4.85 

257.38 

1.91 

1 

6.13 

5.62 

3.57 

2.66 

Project 19 

$389.90 

$26.10 

$159.00 

7.98 

48.86 

1.16 

1 

7.65 

8.62 

4.98 

3.73 

Project 20 

$3,864.70 

$1,257.60 

$1,458.70 

15.46 

249.98 

1.88 

0 

9.48 

9.34 

12.64 

5.21 













Total 

$20,906.60 


$10,263.40 

85.63 



1 To | 

70A6 

69.57 

48.08 

32.32 

Profit/Rank 

$244.15 











ProftScore 

$675,768.93 

Maximize 

<=$10500 

<=100 



x<=10 



<=80 



Figure 3 5. Optimization with Ten (10) Programs 
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Figure 36 is a static efficiency frontier showing the EMV and ROI for the 
different constraint levels. This model was modified to include the percent ROI at the 
different constraint levels. Note that the baseline is annotated in red with an ROI of zero. 


Static Efficiency Frontier 



Comprehensive 

Tactical 

Military Score 

Allowed 

ROI-RANK 

Percent 

Budget 

Score 

Score 

Projects 

Objective 

Change ROI 

$9,500 00 

32.37 

69.03 

70.16 

9 

$689,082.38 

1.97% 

$70,500.00 

32.32 

69.57 

70.46 

10 

$675,768.93 

0.00% 

$11,500.00 

35.82 

72.59 

73.34 

11 

$810,419.62 

19.93% 

$12,500.00 

39.27 

81.77 

80 68 

12 

$984,108.39 

45.63% 

$13,00000 

40 59 

84 01 

82.20 

13 

$998,868.72 

47.81% 


Baseline 


Comprehensive Score 


42.00 

40.00 

38.00 

36.00 

34.00 

32.00 

30.00 



90.00 

85.00 

80.00 

75.00 

70.00 

65.00 



S8.500 S9.000 S9,500 SI0,000 SI0,500 S11,000 S11,500SI2,000 SI2,500SI3,000 SI3,500 


S8.500 S9.000 S9.500 S10,000 510,500 S1 1 ,000 S1 1 ,500 512,000 512,500 S13,000 S13,500 


Tactical Score 


90.00 

85.00 

80.00 

75.00 

70.00 


65.00 



SI ,000,000.00 
5950,000.00 
5900,000.00 
S850.000.00 
S800.000.00 
5750,000.00 
S700,000.00 
S650,000.00 
5600,000.00 


S8.500 S9,000 S9,500 SI0,000 SI0,500S11,000 S11,500 SI2,000 S12,500SI3,000 SI3,500 


58,500 S9.000 S9,500 SI0,000SI0,500SI 1,OOOS11,500S12,OOOS12,500S13,OOOS13,500 


Figure 36. Efficient Frontiers with ROI 






Military Portfolio Optimization 





Project Name 

ENPV 

NPV 

Cost 

Strategy 

Ranking 

Return to 
Rank Ratio 

Profitability 

Index 

Selection 

Military Score 

Tactical 

Score 

FTE 

Resources 

Comprehensive 

Score 

Project 1 

$501.60 

$102.30 

$443.50 

4.26 

117.75 

1.23 

0 

2.51 

1.77 

3.91 

1.40 

Project 2 

$726.30 

$121,30 

$356.60 

5.61 

129.47 

1.34 

0 

3.64 

3.56 

2.73 

1.77 

Project 3 

$932.40 

$354.20 

$925.90 

2.34 

396.46 

1.36 

0 

3.12 

3.24 

5.24 

2.12 

Project 4 

$620.50 

$265.60 

$512.10 

3.21 

193.30 

1.52 

1 

5.14 

5.10 

3.41 

2.46 

Project 5 

$420.60 

$271,20 

$524.50 

1.32 

316.79 

1.52 

0 

3.22 

1.62 

3.29 

1.60 

Project 6 

$2,067.50 

$345.90 

$1,666.70 

4.66 

446.05 

1.19 

0 

6.37 

6.21 

4.56 

2.98 

Project 7 

$3,264.90 

$594.10 

$1,602.10 

10.59 

308.30 

1.37 

1 

6.69 

7.53 

4.23 

3.43 

Project 6 

$2,563.40 

$421.50 

$2,133.40 

2.63 

905.60 

1.20 

1 

2.64 

4.12 

5.61 

2.31 

Project 9 

$5,466.10 

$1,032.20 

$2,712.30 

7.66 

711.99 

1.38 

1 

6.94 

7.99 

4.68 

3.46 

Project 10 

$3,246.90 

$775.40 

$2,502.60 

5.74 

566.01 

1.31 

0 

5.53 

6.23 

6.43 

3.16 

Project 11 

$125.90 

$32.60 

$26.60 

15.26 

6.25 

2.15 

1 

9.84 

9.17 

5.47 

4.13 

Project 12 

$235.10 

$65.60 

$109.50 

6.42 

36.62 

1.76 

0 

4.25 

3.65 

3.91 

2.00 

Project 13 

$4,163.50 

$325.10 

$3,054.20 

5.26 

791.54 

1.11 

0 

3.41 

6.21 

5.23 

2.77 

Project 14 

$4,466.70 

$594.10 

$1,673.60 

10.54 

8.46 

7.54 

1 

8.46 

7.54 

4.23 

3.41 

Project 15 

$567.60 

$421.50 

$409.00 

2.61 

209.11 

2.03 

1 

2.74 

3.52 

5.34 

2.13 

Project 16 

$1,696.40 

$431.90 

$656.30 

11.92 

142.48 

1.66 

1 

6.45 

8.87 

4.26 

3.74 

Project 17 

$756.40 

$134.20 

$465.50 

4.65 

163.10 

1.28 

1 

4.52 

4.35 

6.32 

2.60 

Project 16 

$1,246.30 

$301.50 

$331.50 

4.65 

257.38 

1.91 

1 

6.13 

5.62 

3:57 

2.66 

Project 19 

$369.90 

$26.10 

$159.00 

7.96 

46.86 

1.16 

1 

7.65 

8.62 

4.96 

3.73 

Project 20 

$3,664.70 

$1,257.60 

$1,456.70 

15.46 

249.98 

1.66 

1 

9.48 

9.34 

12.64 

5.21 













Total 

$25,056.60 


$12,164.30 

97.76 



1 « 1 

SOM 

81.7 7 

64.74 

39.27 

Profit/Rank 

$256.26 











Profit*Score 

$964,106.29 

Maximize 

<=$12500 

<=100 



x<=12 



<=60 



Figure 37. Optimal Solution Budget with Twelve Programs 
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Figure 37 shows the optimal solution based on the efficient frontier and ROI. 
This solution will be covered in detail in the analysis section. 

3. Real Options Valuation’s Risk Simulator Data Analysis 

The analysis of the risk simulator model shows, without a doubt, that there is a 
definite efficient frontier in which there is a substantial ROI limit in the profile used in 
this model. As shown in Figure 38 below, from $9,500 to $10,500 there is actually a 
decrease in the ROI maximization objective. From $10,500 to $12,500 there is a 
substantial increase in the EMV and ROI objectives with a rapid slow down in ROI 
above that threshold. Note that by increasing the budget by $2,000 and allowing for 
additional programs, decision makers are given the “opportunity” to increase the Defense 
capabilities outlined in the NDS. This “opportunity” is one of the options available to 
Defense decision makers. The option does not have to be exercised, but it is available if 
needs dictate and funding is available from Congress. Other options include using eleven 
programs or thirteen programs, if funding becomes available, or even using nine 
programs in the case of budget cuts. Note that using nine programs provides an even 
better ROI than the baseline. 


Static Efficiency Frontier 



Comprehensive 

Tactical 

Military Score 

Allowed 

ROPRANK 

Percent 

Burjgef 

Score 

Score 

Projects 

Objective 

Change ROI 

$9,500M 

32.37 

69.03 

70.16 

9 

$689 : 082.38 

1.97% 

610 .. 500.00 

32.32 

69.57 

70.46 

10 

6675.766.93 

0.00% 

6 11,500.00 

35.62 

72.59 

73.34 

11 

$810419. 62 

19.93% 

612..50Q.0Q 

39.27 

6177 

60.66 

12 

$984..m.39 

45.63% 

$12,000.00 

40.59 

84.01 

62.20 

13 

6996.666.72 

47.31% 


Figure 38. Efficient Frontier ROI from Baseline 


This tells the decision makers in Washington that, based on the NDS, a budget of 
$10,500 is not the most optimal to provide for our Defense needs. If more funding were 
allotted, the most optimal solution, based on strategic value, would be a budget of 
$12,500 and twelve programs. Anything more would be a waste of taxpayers’ money by 
funding a low ROI, and anything less would decrease the value of the Defense plan and 
make the budget less effective, while decreasing Defense capabilities. 
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Figure 39 shows the programs that were evaluated by name. The programs in 
green are programs selected at both the ten and twelve program optimization levels, while 
the programs in orange were specifically from the ten program optimization, and the 
programs in blue were from the twelve program optimization. 







Strategy ' 

Military 

Tactical 

FTE 


Program 

ENPV 

NPV 

Cost 

Ranking 

Score 

Score 

Rescue res 

PI 

CH47 Helicopter 

' 5501.60 

$102.30 

$443.50 

1.94 

2.51 

1.77 

3.91 

P2 

Armed Reconnaissance Helicopter 

5726.30 

$121.30 

$358.80 

3.75 

3.84 

3.56 

2.73 

P3 

Blackhawk Helicopter 

1 5932.40 

$354.20 

$925.90 

2.34 

3.12 

3.24 

5.24 

P4 

Patriot Missle System 

" 5620.50 

$265.80 

$512.10 

1.36 

5.14 

5.1 

3.41 

P5 

Patriot Modifications 

" 5420.80 

$271.20 

$524.50 

1.32 

3.22 

1.62 

3.29 

P6 

F/A 18 E/F Fighter 

' 52,087.50 

$345.90 

$1,868.70 

4.68 

6.37 

6.21 

4.56 

P7 

Joint Strike Fighter 

' 53,264.90 

$594.10 

$1,602.10 

10.59 

8.69 

7.53 

4.23 

PS 

V22 Osprey 

' 52,563.40 

$421.50 

$2,133.40 

2.83 

2.64 

4.12 

5.61 

P9 

Carrier Replacement Program 

" 55.468.10 

$1,032.20 

$2,712.30 

7.68 

6.94 

7.99 

4.68 

Flu 

DDG-1000 Program 

' 53.248.90 

$775.40 

$2,502.80 

5.74 

5.53 

6.23 

6.43 

P11 

Marine EOD Systems 

' 5125.90 

$32.80 

$28.60 

15.26 

9.84 

9.17 

5.47 

F1 2 

Marine High Mobility Rocket System^ $235.10 

$85.60 

$109.50 

6.42 

4.25 

3.65 

3.91 

P13 

Air Force F-22 Fighter 

' 54.163.50 

$325.10 

$3,054.20 

5.26 

3.41 

6.21 

5.23 

P14 

Air Force Joint Strike Fighter 

" 54.468.70 

$594.10 

$1,673.80 

10.54 

8.46 

7.54 

4.23 

PIE- 

V22 Osprey 

" 5587.60 

$421.50 

$409.00 

2.81 

2.74 

3.52 

5.34 

PIG 

Global Hawk Remote UAV 

" 51.698.40 

$431.90 

$658.30 

11.92 

8.45 

8.87 

4.26 

P17 

C5A Cargo Plane 

' 5758.40 

$134.20 

$485.50 

4.65 

4.52 

4.35 

6.32 

P18 

C17 Cargo Plane 

' 51,248.30 

$301.50 

$331.50 

4.85 

6.13 

5.62 

3.57 

PI 9 

Defense Space Recon Program 

" 5389.90 

$26.10 

$159.00 

7.98 

7.65 

8.62 

4.98 

P2D 

Special Operations Command 

" 53,864.70 

$1,257.60 

$1,458.70 

15.46 

9.48 

9.34 

12.64 






Figure 39. Real Programs by Name 


Infigure 39, it is interesting to note that at the $10,500 level the program selected 
the DDG-1000 program and the Marine High Mobility Rocket System in addition to the 
other programs in green. This is based on the limited budget and EMVs utilized. At the 
other end of the spectrum, we have included more Patriot Missiles, Air Force and Marine 
V-22’s and additional Special Operations personnel. It is obvious that the increased cost 
was one of the reasons they were not in the initial evaluation, but they bring a higher ROI 
to the Defense capability requirements. Additionally, it should be noted that both the 
DDG (Guided Missile Destroyer) program and the Marine High Mobility Rocket System 
have been dropped. In examining the data, it is obvious that the DDG program was too 
expensive and that the Marine High Mobility Rocket System did not provide a high 
enough return to be included in the Defense budget. 

The programs that made the cut during both evaluations, the Joint Strike Fighter 
and the Global Hawk Unmanned Aerial Vehicle system, made the cut based on their 
capabilities rating and ability to meet the future needs of the Defense Department. 
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Programs not selected in either evaluation were either cost prohibitive or did not provide 
the capabilities required to meet security requirements. Note: The modeling data for 
@Risk is provided in Appendix B. 


59 



THIS PAGE INTENTIONALLY LEFT BLANK 


60 



III. RECOMMENDATIONS 


A. GAO 

The GAO recommends that the Secretary of Defense implement an enterprise¬ 
wide portfolio management approach to making weapon system investments that 
integrates the assessment and determination of war fighting needs with available 
resources and cuts across the services by functional or capability area (GAO, 2008). To 
ensure the success of such an approach, the Secretary should establish a single point of 
accountability at the Department level with the authority, responsibility, and tools to 
ensure that portfolio management for weapon system investments is effectively 
implemented across the Department. In addition, the Secretary should ensure that the 
following commercial best practices, identified in this report, are incorporated: 

• Implement a review process in which needs and resources are integrated 
early and in which resources are committed incrementally based on the 
achievement of specific levels of knowledge at established decision 
points; 

• Prioritize programs based on the relative costs, benefits, and risks of each 
investment to ensure a balanced portfolio; 

• Require increasingly precise cost, schedule, and performance information 
for each alternative that meets specified levels of confidence and 
allowable deviations at each decision point leading up to the start of 
product development; 

• Establish portfolio managers who are empowered to prioritize needs, make 
early go/no-go decisions about alternative solutions, and allocate resources 
within fiscal constraints; and 

• Hold officials at all levels accountable for achieving and maintaining a 
balanced, joint portfolio of weapon system investments that meet the 
needs of the war fighter within resource constraints. 

The GAO also recommends that the Secretary take steps to support Department- 
level decision makers and portfolio managers by developing a stronger joint analytical 
capability to assess and prioritize war fighting needs. 
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B. GARTNER GROUP 


Gartner Group recommendations for end users implementing PA solutions 
include the following: 

• Organizations seeking improved PPM should identify organizational and 
individual “readiness” to adopt the discipline of portfolio management, 
through a survey of capabilities and skills that are “foundational” to the 
use and application of portfolio management, before exploring which tools 
can best support and enhance PPM capabilities. In this way, they can help 
ensure that such projects are successful. 

• Organizations should insist on modular, progressive implementations that 
will fit their immediate requirements, as well as supporting them as they 
evolve over time. 

• Organizations should be prepared to invest in ongoing consulting support 
to ensure that the software is accepted by all stakeholders after 
implementation. 

• Larger vendors will continue to be best-positioned to sustain themselves 
by cross-selling from large software portfolios to installed bases. These 
vendors’ capability to reach down market to small and midsize businesses 
through established channel partners will increase. The market will see 
increased competition from alternative solutions, such as open source and 
Software as a Service, as many vendors expand product offerings and 
collide with other vendors. 

C. RESEARCH TEAM 

The authors of this paper offer the following recommendations: 

• The PA toolsets are needed to make objective comparisons when working 
with choices containing numerous or complex input variables and 
unknown risk. As such, these tools are essential for project and program 
managers who are working with a multitude of projects or programs. 
Recommendation: NPS should explore the interest and potential for 
developing (or purchasing) a program management level curriculum for 
choosing and using PA and PM tools. The point of this curriculum would 
be to educate and indoctrinate future program managers in how to make 
defense management decisions based on acceptable risk levels utilizing 
PM guidance and PA decision making tools. 

• The choices of PA tools are numerous, and selection criteria can be varied 
across organizations. As PA tools proliferate through DoD organizations 
there is a potential for implementation of a wide variety of (proprietary) 
PA tool suites, which would lead to costly support and interoperability 
issues. Recommendation: NPS should explore the potential for 
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development of a PA/PM Lab to be used for product evaluation, source 
selection criteria development, and making recommendations to defense 
officials on which products to support. 

• A DoD umbrella organization is needed to establish policies, guidelines, 
best practices, and provide PA implementation planning assistance. 
Recommendation: Building upon the PA Lab concept in Recommendation 
“b”, develop an NPS/PA Center of Excellence (CoE) in Monterey. This 
CoE would serve to coordinate and educate PA practitioners and provide a 
“Net Centric” environment to tie Joint PA efforts together. The CoE is 
part of the solution to satisfy the GAO recommendation: “The GAO also 
recommends that the Secretary take steps to support department-level 
decision makers and portfolio managers by developing a stronger joint 
analytical capability to assess and prioritize warfighting needs.” 

In summary, the recommendations are: 1) develop PA curriculum and education 
programs, 2) develop a PA solutions lab, and 3) develop a PA CoE. 
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IV. CONCLUSIONS 


A. COMPARING THE EVALUATION MODEL WITH REALITY 


After developing the evaluation model, running simulations, and performing data 
analysis, the research team has concluded that Risk Simulator is a very capable Microsoft 
Excel plug-in which can evaluate program risk, evaluate EMVs, and optimize budgeting 
and programming constraints, all within the scope of the NDS. This software also has 
hundreds of risk models built in, including the military model. These models are easily 
modifiable to fit any number or programming requirements. To verify this assessment, 
refer to Figure 40, the original model with the real program names. 



Program 

ENPV 

NPV 

Cost 

Strategy ’ 
Ranking 

Military 

Score 

Tactical 

Score 

- FTE -- 

Rescue res 

PI 

CH47 Helicopter 


$501.60 

$102.30 

$443.50 

1.94 

2.51 

1.77 

3.91 

P2 

Armed Reconnaissance Helicopter 


$726.30 

$121.30 

$358.80 

3.75 

3.84 

3.56 

2.73 

P3 

Blackhawk Helicopter 


$932.40 

$354.20 

$925.90 

2.34 

3.12 

3.24 

5.24 

P4 

Patriot Missle System 


$620.50 

$265.80 

$512.10 

1.36 

5.14 

5.1 

3.41 

P5 

Patriot Modifications 


$420.80 

$271.20 

$524.50 

1.32 

3.22 

1.62 

3.29 

P6 

F/A 18 E/F Fighter 


$2,087.50 

$345.90 

$1,868.70 

4.68 

6.37 

6.21 

4.56 

P7 

Joint Strike Fighter 


$3,264.90 

$594.10 

$1,602.10 

10.59 

8.69 

7.53 

4.23 

PS 

V22 Osprey 


$2,563.40 

$421.50 

$2,133.40 

2.83 

2.64 

4.12 

5.61 

PS 

Carrier Replacement Program 


$5,468.10 

$1,032.20 

$2,712.30 

7.68 

6.94 

7.99 

4.68 

P10 

DDG-1000 Program 


$3,248.90 

$775.40 

$2,502.80 

5.74 

5.53 

6.23 

6.43 

P11 

Marine EOD Systems 


$125.90 

$32.80 

$28.60 

15.26 

9.84 

9.17 

5.47 

P12 

Marine High Mobility Rocket System 1 

$235.10 

$85.60 

$109.50 

6.42 

4.25 

3.65 

3.91 

P13 

Air Force F-22 Fighter 


$4,163.50 

$325.10 

$3,054.20 

5.26 

3.41 

6.21 

5.23 

P14 

Air Force Joint Strike Fighter 


$4,468.70 

$594.10 

$1,673.80 

10.54 

8.46 

7.54 

4.23 

P15 

V22 Osprey 


$587.60 

$421.50 

$409.00 

2.81 

2.74 

3.52 

5.34 

PIS 

Global Hawk Remote UAV 


$1,698.40 

$431.90 

$658.30 

11.92 

8.45 

8.87 

4.26 

P17 

C5A Cargo Plane 


$758.40 

$134.20 

$485.50 

4.65 

4.52 

4.35 

6.32 

PIS 

C17 Cargo Plane 


$1,248.30 

$301.50 

$331.50 

4.85 

6.13 

5.62 

3.57 

P19 

Defense Space Recon Program 


$389.90 

$26.10 

$159.00 

7.98 

7.65 

8.62 

4.98 

P20 

Special Operations Command 


$3,864.70 

$1,257.60 

$1,458.70 

15.46 

9.48 

9.34 

12.64 


Figure 40. Evaluation Model with Real Program Names 


On 6 April 2009, Secretary of Defense Gates held a press conference outlining the 
2010 Defense Budget request. To the astonishment of Congress, the Defense Industrial 
Complex, and defense contractors it was announced that he planned to cut the F-22 
fighter program, stop building the DDG-1000, increase production of the Joint Strike 
Fighter and increase the number of Special Operations personnel. These 
recommendations are directly in line with the evaluation results produced by Real 
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Options Valuation’s Risk Simulator. Although there are some differences in the actual 
budget request and our evaluation, they would be consistent if we had known the EMV 
scoring used by the Secretary. 

B. MODELING USING COTS PRODUCTS 

Gartner and Forrester identified several PPM software suites which can be used to 
develop and manage project models. However, these models do not offer robust 
simulations which account for a range of probability distributions while accounting for 
risk across model scenarios. These products excel at providing graphical representations 
of complex data in the form of digital “dashboards,” bubble-charts, and efficient 
frontiers. When used in conjunction with PA modeling software, these PPM suites are 
excellent at helping to efficiently manage large projects while helping to mitigate risk. 

The authors of this paper found that the Real Options Valuation’s Risk Simulator 
and @RISK were better suited for the research being conducted in accordance with the 
intent of this research paper. Risk Simulator was by far the superior product evaluated 
and provided hundreds of readymade models including; Military models, Efficient 
Frontier Models, multiple simulations. These modeling tools leverage the capabilities of 
Microsoft Excel and Monte-Carlo simulation to develop a range of statistical probability 
distributions using an array of variable inputs. This provides the ability to look at the 
best, worst and most likely scenarios. 

C. RESEARCH LIMITATIONS 

The authors found that the “boil the ocean” approach to evaluating products was 
beyond the scope that was executable in a (1) distributed team environment (i.e., team not 
collocated), and (2) without a central laboratory environment with configuration control 
of the products being evaluated. Additionally, it was difficult to codify all the numerous 
requirements and variables from multiple sources. The model was developed using only 
a few of the variables and requirements identified in Table 2 . More complex models 
could be developed using the work in this document as a starting point. 
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APPENDIX A. KEY REQUIREMENTS SOURCES 


Table 3. Key Requirements Sources 


Requirements 

Source 

Author 

Key Requirements 

1. Dr. Flynn’s 
research paper, 
brief, and Joint 
Applied Project 
Prompt 

Dr. Flynn 

I. The 3 major goals of portfolio analysis as provided by 
Dr. Flynn are: 

a) Value Maximization (focus is ROI) 

b) Balance - Short-term versus long-term, High-risk versus 
low-risk, across product categories, Basic research vs. 
production 

c) Strategic Direction - “On-strategy” 

II. The major portfolio analysis requirements (top level) 
that have been identified are: 

1. Baseline Requirements: 

a) Gather Life-Cycle cost data 

b) Establish Scoring System that determine how effectively 
systems match capabilities to requirements 

c) Develop way to display results to allow for both risk- 
reward analysis and trade-off and establish a Scoring 

System 

2. Process Requirements to follow during analysis: 

a) Maintain cognizance of the following inputs or 
constraints during Portfolio Analysis (see spreadsheet for 
particulars) 

b) Parameters to consider for achieving the Portfolio 

Analysis Goal of BALANCE (see spreadsheet for 
particulars) 

c) Requirements during the process of Maximizing Value 
phase (see spreadsheet for particulars) 

d) Scorecard process/contents requirements (see spreadsheet 
for particulars) 

2. DoDD 7450.20 
directive 

mandating use of 
PA in DoD 

systems 

SECDEF 
reference (c) 

I. Use the Joint Capability Area common framework 
and lexicon for the organization of capability portfolios 

II. Cover material and non-material investments 

III. Leverage operational expertise of Combatant 
Commands 


71 







Requirements 

Source 


Author 


3. GAO Report on GAO 
PA Best Practices reference 


Key Requirements 


Use of risk/reward “bubble diagram 


(b) 


Figure 4: Risk Versus Rewards'Matrix 



4. Use of PA in 
Logistics Planning 
Scenarios 


Defense 
Acquisition 
University 
reference (d) 


Classify every purchase or family of purchases into one of 
four categories or quadrants: acquisition , multiple , leverage , 
or strategic ; 


at 

50 


Acquisition 

Few suppliers 
Low value 


Multiple 

Many suppliers 
Low value 


Strategic 
Few suppliers 
High value 


Leverage 
Many suppliers 
High value 


Item Criticality or Value 
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Author 


Requirements 

Source 


Key Requirements 


5. Portfolio 


RAND 


Schematic Depiction of Finding Points Near the Efficient Frontier 


Analysis Methods 


reference (i) 


Efficient frontier 



6. Portfolio 


RAND 


RAND 


Capabilities Versus Time 


Analysis Methods 


reference (i) 



Size of attack 


Geography 


Attack tactics 


Countermeasures 


Strategic warning 


— Near to mid-term 
Long term 

—A— Very long term 


7. Project 
Management Body 
of Knowledge - 
Functional 
Dimensions 


Project 

Management 

Institute 


I. Time management — Manages deliverable activity 
timelines and deadlines for programs, projects, tasks and 
assignments 

II. Resource management — Manages allocation of 
available personnel using a resource profile repository, and 
allows resource loading and leveling (natively, or at 
minimum via third-party tool integration) 


III. Cost management — Tracks resource (and, often, 
other) costs and facilitates chargeback or billing of project 
expenses; for example, those associated with time, travel, 
equipment or other material 
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Requirements 

Source 

Author 

Key Requirements 



IV. Integration (portfolio) management — Provides 
business intelligence to improve portfolio alignment, 
planning, and project or service delivery. Dashboard tools 
provide an integrated view of program and project status, 
resource capacity, service levels, strategic alignment, and 
more. Integration can also be to back-office systems. 

Gathers portfolio data from these segments to let users 
manage their pipelines, report and forecast project progress, 
and perform portfolio analysis and prioritization. 

8. “Assessing 

Risks and 

Returns: A Guide 
for 

Evaluating 

Federal Agencies’ 
IT Investment 
Decision-Making,” 

GAO 

reference (q) 
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9. “Assessing 

Risks and 

Returns: A Guide 
for 

Evaluating 

Federal Agencies’ 
IT Investment 
Decision-Making,” 

GAO 

reference (q) 

B. ORGANIZATIONAL IMPACT (relative weight = 10 
points) Measures the impact n organizational personnel of 
the system. The more favorable the impact on the 
organization the higher its score. 

B.l. Personnel and Training (3 of 10 points) Assess the 
impact of the system on the knowledge, skill, and training 
of organizational personnel if the system in implemented. 
Score from zero to three based on the scale below: 

Zero Points: System is likely to require significant new 
skills to operate and support and the project does not appear 
to mitigate this impact through appropriate training, changes 
in rating qualifications, etc. 

Three Points: System is an improvement to an existing 
system and will require relatively little new skill and/or 
knowledge to operate or support. If it is a new system, it 
will introduce valuable new skills and knowledge to the 
organization and the project will mitigate any adverse 
impact through appropriate training, planning for rating 
qualification changes, etc. 
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Author 


Requirements 

Source 


Key Requirements 


B.2. Scope of Beneficiaries/ Cross-Functional (4 of 10 

points). Assess a higher score (zero to four) the broader the 
scope of beneficiaries. 

Zero Points: Limited number of beneficiaries. This system 
will be used by only one office in headquarters, a single 
area, or district. Not a cross-functional system. 

Four Points: System is cross-functional and serves a number 
of offices, areas, and/or districts. System will be used by a 
large number of organizational units. System will be used 
by the public. 

B. 3. Quality of Work Life (3 of 10 points). Measures the 
improvement in quality of work life expected for the 
systems. Score higher (zero to three points) the more work 
life improvement is expected. 

Zero Points: Little if any positive impact on the quality of 
work life. System may increase the work required (e.g., 
additional data entry). 

Three Points: Positive contribution to the quality of work 
life will clearly result. For example, the system will improve 
medical care for dependents or allow a job to be done much 
faster such that job satisfaction will increase. 

C. MISSION EFFECTIVENESS (relative weight = 20 
points). Measures the impact of the system on both external 
and internal customers. It is a measure of the system’s 
ability to improve the performance of support or operational 
programs. This improvement should be measured in 
quantitative terms, but not in dollars. The economic (dollar) 
impact is captured in the benefit/ cost ratio. However, the 
same benefits might be measured here in a different manner. 
For example, improvements might be expressed in terms of 
accomplishing a task sooner (hours or minutes), delivering a 
service with fewer mistakes, increasing the availability of a 
computer system for customer use (hours per month saved 
in time for system backups), or a number of similar terms. 
The more the project or system improves mission 
effectiveness the higher the score. 

C.l. Improve Internal Program Services (10 of 20 

points). Assess the expected improvement in service to 
internal customers. For example the system might improve 
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Author 


Requirements 

Source 


Key Requirements 


the timeliness of financial reporting throughout the 
organization. Score (zero to ten) higher, the more that 
service will be improved in response to a problem expressed 
by users of the service. 


Zero Points: System does not appear to meet a problem 
defined by an internal customer. Little improvement in 
important customer service criteria, such as timeliness, 
quality, or availability is expected. An improvement is 
described but not quantified. 


Ten Points: A significant improvement expected in areas 
such as timeliness, quality, or availability and the 
improvement is quantified. The improvement also addresses 
an important problem or area of service improvement 
defined by the customer. 


C.2. Improved Service to the Public (10 of 20 points). 
Assess the expected improvement in service to the public. 
Score (zero to ten) higher, the more improvement is 
anticipated in response to a requirement defined by the 
public. 


Zero Points: System appears to provide little or no direct 
improvement in service to the public. Systems may make a 
small improvement in timeliness, quality, or availability, but 
there is no documented need for such an improvement. The 
improvement is not quantified. 


Ten Points: System significantly improves service to the 
public in a mission where need is demonstrated or provides 
a new type of service to meet changing customer demands. 
The improvement is quantified. 


D. STRATEGIC ALIGNMENT (relative weight = 25 
points). Measures to what extent the proposed investment 
supports strategic organizational objectives. Scoring is 
based primarily on explicit documentation of the need for 
the Integrated Risk Management (IRM) system in planning 
documents. The more the project or system is aligned with 
program/strategic goals, the higher the score. 

D.l. Business Model (7 of 25 points). Assess the degree of 
alignment with the organization’s business model. 
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Author 


Requirements 

Source 


Key Requirements 


Zero Points: Proposed project or system does not support 
organizational products/services or processes identified in 
the business model. 


One to Four Points: Proposed system is specifically 
mentioned in the 5-year IRM plan and supports 
organizational products/services or processes identified in 
the business model. (Score one to two points if the system 
supports products/services or processes in the business 
model, but is not listed in the 5-year IRM plan.) 

Five to Seven Points: Proposed system is specifically 
mentioned in the 5-year IRM plan and supports 
products/services or processes identified in the business 
model, and the project has been coordinated with all offices 
identified by the model for the respective processes the 
system supports. 

D.3. Business Process Redesign (6 of 25 points). Assess 
the degree this system enables the organization to do 
business in a better way. Score (zero to six) higher the 
greater the expected improvement. 

Zero Points: This system automates an existing business 
process with little improvement of the process (i.e., helps to 
do the same thing faster). 

Six Points: System enables a significant improvement in the 
way business is conducted 

D. 2. Level of Interest (12 of 25 points). Assess the level of 
interest by senior managers (at agency and departmental 
level) and/or the Congress. Score (zero to twelve) higher the 
greater the level of interest. 

Zero Points: No expressed support for this system by senior 
managers or the Congress. 

Twelve Points: System has strong support from the 
Congress, departmental senior managers, and/or the head of 
the agency. System is specifically mentioned in 
determinations. 

E. BENEFIT-COST IMPACT(S) (relative weight = 25 
points). Measures the value of the system in dollar terms. 
The system benefit/cost ration is the key indicator. This 
ration is developed using the standard benefit-cost guidance 
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Author 


Requirements 

Source 


Key Requirements 


and spreadsheet promulgated by the organization. The 
standard guidance ensures all system studies include a 
common set of costs and approach benefits definition in a 
similar manner. The standard spreadsheet assists system 
sponsors in the benefit/cost calculation. The higher the 
benefit/cost ratio, the better the score. 

Zero Points: Any benefit/cost ratio less than one (i.e., costs 
exceed the benefits). 

One Point: Benefit/cost ratio of one. 

Five Points: Benefit/cost ratio of 1.5 to 1.75. 

Ten Points: Benefit/cost ratio of 1.76 to 1.99. 

Fifteen Points: Benefit/cost ratio of 2.0 to 2.99. 

Twenty Points: Benefit/cost ratio of 3.0 to 3.99. 

Twenty-five Points: Benefit/cost ratio of 4.0 or greater 
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APPENDIX B. PALISADE @RISK MODELING DATA 

AND ANALYSIS 


A. PALISADE @RISK MODELING DATA 

In this model, the user provides the Project Name, the Budget for the project, and 
then provides Minimum, Most Likely (ML) and Maximum budget values. The individual 
project percentages should reflect of the actual cost associated with the project as 
represented by the ML and Maximum budget overruns expected. This model could be 
enhanced with additional variables representing project risk, or ranges of risk represented 
as Min/ML/Max values. The user may change any of the light green shaded cells, which 
are the model’s inputs; the significant model outputs are shaded in light blue. 

1. @RISK Model using Minimum, Most Likely (ML) and Maximum 
Budget Values 


Table 4. Budget Simulation 


Project 

Budget 

Min 

ML 

Max 

Min$ 

ML$ 

Max$ 

Budget 

Simulation 

1000 Iterations 

Project 1 

$1,732.44 

100% 

110% 

125% 

$ 

1,732.44 

$ 

1,905.68 

$ 2,165.55 

$ 

1,920.12 

Project 2 

$859.00 

100% 

110% 

125% 

$ 

859.00 

$ 

944.90 

$ 

1,073.75 

$ 

952.06 

Project 3 

$1,845.00 

100% 

110% 

125% 

$ 

1,845.00 

$ 

2,029.50 

$ 

2,306.25 

$ 

2,044.88 

Project 4 

$1,645.00 

100% 

110% 

125% 

$ 

1,645.00 

$ 

1,809.50 

$ 

2,056.25 

$ 

1,823.21 

Project 5 

$458.00 

100% 

110% 

125% 

$ 

458.00 

$ 

503.80 

$ 

572.50 

$ 

507.62 

Project 6 

$52.00 

100% 

110% 

125% 

$ 

52.00 

$ 

57.20 

$ 

65.00 

$ 

57.63 

Project 7 

$758.00 

100% 

110% 

125% 

$ 

758.00 

$ 

833.80 

$ 

947.50 

$ 

840.12 

Project 8 

$115.00 

100% 

110% 

125% 

$ 

115.00 

$ 

126.50 

$ 

143.75 

$ 

127.46 

Project 9 

$125.00 

100% 

110% 

125% 

$ 

125.00 

$ 

137.50 

$ 

156.25 

$ 

138.54 

Project 10 

$458.00 

100% 

110% 

125% 

$ 

458.00 

$ 

503.80 

$ 

572.50 

$ 

507.62 

Project 11 

$45.00 

100% 

110% 

125% 

$ 

45.00 

$ 

49.50 

$ 

56.25 

$ 

49.88 

Project 12 

$105.00 

100% 

110% 

125% 

$ 

105.00 

$ 

115.50 

$ 

131.25 

$ 

116.38 

Project 13 

$48.00 

100% 

110% 

125% 

$ 

48.00 

$ 

52.80 

$ 

60.00 

$ 

53.20 

Project 14 

$351.00 

100% 

110% 

125% 

$ 

351.00 

$ 

386.10 

$ 

438.75 

$ 

389.03 

Project 15 

$421.00 

100% 

110% 

125% 

$ 

421.00 

$ 

463.10 

$ 

526.25 

$ 

466.61 

Project 16 

$124.00 

100% 

110% 

125% 

$ 

124.00 

$ 

136.40 

$ 

155.00 

$ 

137.43 

Project 17 

$521.00 

100% 

110% 

125% 

$ 

521.00 

$ 

573.10 

$ 

651.25 

$ 

577.44 
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Project 

Budget 

Min 

ML 

Max 

Min$ 

ML$ 

Max$ 

Budget 

Simulation 

1000 Iterations 

Project 18 

$512.00 

100% 

110% 

125% 

$ 

512.00 

$ 

563.20 

$ 

640.00 

$ 

567.47 

Project 19 

$5.00 

100% 

110% 

125% 

$ 

5.00 

$ 

5.50 

$ 

6.25 

$ 

5.54 

Project 20 

$21.00 

100% 

110% 

125% 

$ 

21.00 

$ 

23.10 

$ 

26.25 

$ 

23.28 

Total 

$10,200.44 




$ 

10,200.44 

$ 11,220.48 

$ 12,750.55 

$ 

11,305.48 


The distribution curve for the budget simulation is shown in Table 4 above. This 
model uses the previous results from 1000 simulations to arrive at a 90% probability that 
the projects can be executed at a budget cost (Simulated Budget) of $11,514, or a delta 
cost increase of $1,314 from the budget baseline of $10,200 (Figure 41). 
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Figure 41. Cost Probability Distribution 
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Building upon the results of the previous model, in the next model, Table 5, the 
Project Name and the Mean Budget Simulation are inherited from the previous model. 
New columns for Strategic Value, Tactical Value and FTE Resources are added. These 
columns are scores (which would be) provided by a panel of military experts which can be 
considered a Delphi panel; for this model, the scores were determined by the authors. The 
simple addition of these the Delphi scores are added to create an overall KVA score. The 
EMV is then calculated by dividing the KVA with the simulated budget value. The EMV 
is now akin to a ROI which is used to evaluate and rank commercial projects. Note that 
the Value scores are not weighted in this model. 

2. @RISK Model using Strategic, Tactical and FTE Resource values 


Table 5. Expected Military Value 


Projects 

Budget 

Simulation 

Strategic 

Value 

Tactical 

Value 

FTE 

Resources 

KVA 

Scores 

EMV 

(KVA/Budget) 






11.61 

0.605% 






8.60 

0.903% 






18.23 

0.892% 






14.94 

0.819% 






16.77 

3.303% 






18.63 

32.333% 






24.24 

2.885% 






22.27 

17.469% 






14.37 

10.371% 






17.20 

3.389% 






17.07 

34.216% 






15.83 

13.603% 






9.28 

17.440% 






28.80 

7.403% 

Project 15 





23.73 

5.085% 






24.55 

17.860% 






12.35 

2.138% 






10.29 

1.813% 






10.57 

190.741% 






11.27 

48.421% 

Total 

$ 11,305.48 
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In this model, the values are inherited from the previous two models. The model is 
then sorted in descending sequence by EMV. 

3. @RISK Model Sorted by EMV 

Table 6 below shows the top ranked EMV. The key aspects for the models are the 
expected budget values, and the scores provided by the Delphi panel of experts. The 
validity of the input values is critical to obtaining usable results from these @RISK 
models. 


Table 6. Top Ranked EMV 


Programs 

Budget Simulation 

KVA 

Scores 

EMV 

(KVA/Budget) 

Project 19 

$5.54 

10.57 

190.741% 

Project 20 

$23.28 

11.27 

48.421% 

Project 11 

$49.88 

17.07 

34.216% 

Project 6 

$57.63 

18.63 

32.333% 

Project 16 

$137.43 

24.55 

17.860% 

Project 8 

$127.46 

22.27 

17.469% 

Project 13 

$53.20 

9.28 

17.440% 

Project 12 

$116.38 

15.83 

13.603% 

Project 9 

$138.54 

14.37 

10.371% 

Project 14 

$389.03 

28.80 

7.403% 

Project 15 

$466.61 

23.73 

5.085% 

Project 10 

$507.62 

17.20 

3.389% 

Project 5 

$507.62 

16.77 

3.303% 

Project 7 

$840.12 

24.24 

2.885% 

Project 17 

$577.44 

12.35 

2.138% 

Project 18 

$567.47 

10.29 

1.813% 

Project 2 

$952.06 

8.60 

0.903% 

Project 3 

$2,044.88 

18.23 

0.892% 

Project 4 

$1,823.21 

14.94 

0.819% 

Project 1 

$1,920.12 

11.61 

0.605% 

Total 

$ 11,305.48 
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4. 


Palisade @RISK Data Analysis 


This section provides graphical representations of the preceding data tables, and 
provides analysis of the information. The graphics in Figure 42 and Figure 43 compare 
the original budget values with the simulated budget values following 1,000 Monte Carlo 
simulation iterations. The delta between the budgets (original vs. simulated) is driven by 
the low, most-likely and maximum cost values provided to the model. 
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Figure 42. Budget (Original vs. Simulated #1) 


The graphic in Figure 44 compares the project cost (Sim$) with the EMV for each 
project. Projects where the EMV point is above the Sim$ point are the most favorable 
from an EMV perspective. In this particular simulation, the highest cost projects appear to 
have the least return (based upon the notional weighting applied to the KVA). 
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Figure 43. Budget (Original vs. Simulated #2) 


—»— Sim$ (Millions) EMV 



Project Number 


Figure 44. Project Cost (Sim$) Compared with EMV 
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The graphic in Figure 45 stratifies the EMV low to high. In this simulation, only 
(4) projects (#6, 11, 20, 19) have an EMY greater than 20%. 
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Figure 45. Project EMV Low to High 


While this model seems to provide for a solution to the original model, it in fact 
does not. The different models are not integrated with each other which does not allow for 
a continuous simulation, optimization cycle that is required to create a valid efficient 
frontier. This would be one of the software programs used by the NPS PA CoE to 
evaluate the possibility of being utilized for future implementation, though at this time, it 
is not a viable candidate for evaluating defense acquisition capability requirements. 
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APPENDIX C. CREATION OF AN EFFICIENT FRONTIER 


Using Risk Simulator to create an efficient frontier is easily accomplished. As 
depicted in Figure 46, select any green cell (A) in the model (these are already set up 
Monte Carlo simulation assumptions). Reset or set new assumptions by clicking on Risk 
Simulator, Set Input Assumptions (B) and selecting the desired distribution. In this case, 
select Triangular (C) and enter in the desired Minimum, Likely, and Maximum values 
(C). 



Figure 46. Efficient Frontier Setting Assumptions 

Next, select the desired output. In the example depicted in Figure 47, cell C26 (D) 
is the total NPV for the portfolio (EMV can also be selected if desired). Click on Set 
Output Forecast (E) and give provide a name (F). Run Simulation or Run Super Speed 
(G), which displays chart (H). Set the probability of exceeding or being below some 
value (H). You can then analyze the Statistics tab to look at all the risk factors and risk 
coefficients (e.g., coefficient of variation). You can also see a two tail or left tail 
boundary, as well as adjust the percent of certainty in the chart. 
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Figure 47. Setting the Output 

Now you are ready to run the optimization (I) in Figure 48. Here you can select 
the Static, Dynamic or Stochastic Optimization routine (K). As shown in (K) below, 
simulation is tied into the optimization routines (i.e., these two items cannot be separated 
into two different applications). When optimizing using either the Dynamic or Stochastic 
routines, you will need to simulate, optimize, and then repeat multiple times. 



Figure 48. Starting Optimization 
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As shown in (L) below in Figure 49 you can also set constraints, then run the 
optimization model (M) again to add in the Efficient Frontier (N) by selecting the 
constraints of interest (O) and adding them to the changing constraints list to run the 
efficient frontier. 
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Figure 49. Setting Constraints 


The efficient frontiers generated by the optimization illustrate the best 
combination and permutation of projects in the optimal portfolio. Each point on the 
frontier is a portfolio of various combinations of projects that provide the best allocation 
possible given the requirements and constraints (Mun & Housel, 2006, p. 12). Because 
this is an integrated process, simulated and optimized in sequence, together, the analysis 
will also be integrated, which is why it is called Integrated Risk Management (IRM). 
Since the process is integrated, it cannot be taken apart to allow some parts to be done 
one way and other parts to be done in another way and then merged back together. If the 
analysis is performed in a way that is not integrated, the results will be erroneous. 
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